[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:20 dist-test-slave-k477 krb5kdc[25330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:20 dist-test-slave-k477 krb5kdc[25330](info): set up 2 sockets
Sep 28 20:24:20 dist-test-slave-k477 krb5kdc[25330](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 20:24:21.992290 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 20:24:22.216598 25323 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.144s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 20:24:22.421236 25323 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.204s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/test-user.keytab.
W20260928 20:24:23.354688 25323 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.933s user 0.001s sys 0.000s
Sep 28 20:24:23 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
W20260928 20:24:24.523588 25323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.186.254: real 1.144s user 0.000s sys 0.003s
I20260928 20:24:24.524211 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44633
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:24.620502 25343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:24.620798 25343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:24.620841 25343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:24.622176 25343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:24.622248 25343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:24.622272 25343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:24.622303 25343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:24.622356 25343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:24.624078 25343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:24.624372 25343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:24.624629 25343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:24:24.630733 25343 server_base.cc:1034] running on GCE node
W20260928 20:24:24.630843 25348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:24.630787 25349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:24.630734 25351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:24.631210 25343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:24.631551 25343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:24.632725 25343 hybrid_clock.cc:648] HybridClock initialized: now 1790627064632706 us; error 39 us; skew 500 ppm
Sep 28 20:24:24 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627064, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:24.635423 25343 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:24.635977 25343 webserver.cc:460] Webserver started at http://127.24.186.254:41285/ using document root <none> and password file <none>
I20260928 20:24:24.636240 25343 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:24.636327 25343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:24.636461 25343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:24.637645 25343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "85369a0304964cd29a797bcf21214934"
format_stamp: "Formatted at 2026-09-28 20:24:24 on dist-test-slave-k477"
server_key: "2e9b6d7c35b1de5eb938313a7105a7ca"
server_key_iv: "287ffaa475ceeec39d0a62ee97446c09"
server_key_version: "encryptionkey@0"
I20260928 20:24:24.637979 25343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "85369a0304964cd29a797bcf21214934"
format_stamp: "Formatted at 2026-09-28 20:24:24 on dist-test-slave-k477"
server_key: "2e9b6d7c35b1de5eb938313a7105a7ca"
server_key_iv: "287ffaa475ceeec39d0a62ee97446c09"
server_key_version: "encryptionkey@0"
I20260928 20:24:24.639556 25343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:24.640758 25358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:24.641137 25343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:24.641268 25343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "85369a0304964cd29a797bcf21214934"
format_stamp: "Formatted at 2026-09-28 20:24:24 on dist-test-slave-k477"
server_key: "2e9b6d7c35b1de5eb938313a7105a7ca"
server_key_iv: "287ffaa475ceeec39d0a62ee97446c09"
server_key_version: "encryptionkey@0"
I20260928 20:24:24.641372 25343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:24.651885 25343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:24.654340 25343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:24.654527 25343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:24.659056 25343 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:44633
I20260928 20:24:24.659140 25410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:44633 every 8 connection(s)
I20260928 20:24:24.659548 25343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:24.660386 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25343
I20260928 20:24:24.660507 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:24.660498 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:24.661190 25323 external_mini_cluster.cc:1428] Setting key 04b147561f9bf47493121b105b2f8de0
I20260928 20:24:24.663301 25411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934: Bootstrap starting.
I20260928 20:24:24.663956 25411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:24.664297 25411 log.cc:824] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:24.665088 25411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934: No bootstrap required, opened a new log
I20260928 20:24:24.667188 25411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } }
I20260928 20:24:24.667393 25411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:24.667433 25411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85369a0304964cd29a797bcf21214934, State: Initialized, Role: FOLLOWER
I20260928 20:24:24.667559 25411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } }
I20260928 20:24:24.667644 25411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:24.667675 25411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:24.667719 25411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 20:24:24 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:24.677485 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:24.662081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.24.186.254:44633)
0928 20:24:24.662282 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:24.662292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:24.662496 (+ 204us) server_negotiation.cc:407] Connection header received
0928 20:24:24.664781 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:24.664790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:24.664851 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:24.665003 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:24.666531 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:24.668185 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:24.669395 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:24.669891 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:24.669981 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:24.673296 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:24.673301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:24.673301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:24.673312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:24.675040 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:24.675454 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:24.675456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:24.675457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:24.675576 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:24.675969 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:24.675972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:24.675973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:24.676450 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:24.676727 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:24.677102 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 20:24:24.677298 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260928 20:24:24.708599 25411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } }
I20260928 20:24:24.708771 25411 leader_election.cc:302] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85369a0304964cd29a797bcf21214934; no voters:
I20260928 20:24:24.709050 25411 leader_election.cc:288] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:24.709133 25416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:24.709340 25416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [term 1 LEADER]: Becoming Leader. State: Replica: 85369a0304964cd29a797bcf21214934, State: Running, Role: LEADER
I20260928 20:24:24.709378 25411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:24.709478 25416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } }
I20260928 20:24:24.710253 25418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85369a0304964cd29a797bcf21214934. Latest consensus state: current_term: 1 leader_uuid: "85369a0304964cd29a797bcf21214934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } } }
I20260928 20:24:24.710451 25418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:24.710472 25417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85369a0304964cd29a797bcf21214934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85369a0304964cd29a797bcf21214934" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44633 } } }
I20260928 20:24:24.710587 25417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:24.710902 25425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:24.711586 25425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:24.713519 25425 catalog_manager.cc:1347] Generated new cluster ID: 514a34e8bf7b4d3e9eac70bedbfd023a
I20260928 20:24:24.713583 25425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:24.735786 25425 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:24.736608 25425 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:24.739990 25425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85369a0304964cd29a797bcf21214934: Generated new TSK 0
I20260928 20:24:24.740223 25425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
W20260928 20:24:25.693147 25323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.186.193: real 0.945s user 0.003s sys 0.001s
I20260928 20:24:25.693533 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:25.785898 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:25.786218 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:25.786271 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:25.788130 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:25.788223 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:25.788272 25437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:25.789971 25437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:25.790283 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:25.790552 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:25.796968 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:25.796977 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:25.796977 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:25.797377 25437 server_base.cc:1034] running on GCE node
I20260928 20:24:25.797654 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:25.798007 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:25.799355 25437 hybrid_clock.cc:648] HybridClock initialized: now 1790627065799314 us; error 41 us; skew 500 ppm
Sep 28 20:24:25 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627065, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:25.802712 25437 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:25.803443 25437 webserver.cc:460] Webserver started at http://127.24.186.193:40569/ using document root <none> and password file <none>
I20260928 20:24:25.803685 25437 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:25.803731 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:25.803906 25437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:25.805068 25437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "cd26575c2a3b4b9ab8f003c3fe569717"
format_stamp: "Formatted at 2026-09-28 20:24:25 on dist-test-slave-k477"
server_key: "f0e7a6cbc33b0af77cd1eff5456dd73c"
server_key_iv: "63477361fcab3c2a8f231fe78bcad431"
server_key_version: "encryptionkey@0"
I20260928 20:24:25.805418 25437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "cd26575c2a3b4b9ab8f003c3fe569717"
format_stamp: "Formatted at 2026-09-28 20:24:25 on dist-test-slave-k477"
server_key: "f0e7a6cbc33b0af77cd1eff5456dd73c"
server_key_iv: "63477361fcab3c2a8f231fe78bcad431"
server_key_version: "encryptionkey@0"
I20260928 20:24:25.807204 25437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:25.808223 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:25.808511 25437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:25.808624 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "cd26575c2a3b4b9ab8f003c3fe569717"
format_stamp: "Formatted at 2026-09-28 20:24:25 on dist-test-slave-k477"
server_key: "f0e7a6cbc33b0af77cd1eff5456dd73c"
server_key_iv: "63477361fcab3c2a8f231fe78bcad431"
server_key_version: "encryptionkey@0"
I20260928 20:24:25.808709 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:25.825938 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:25.828444 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:25.828601 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:25.828872 25437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:25.829203 25437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:25.829267 25437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:25.829331 25437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:25.829360 25437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:25.835314 25437 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:46769
I20260928 20:24:25.835357 25565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:46769 every 8 connection(s)
I20260928 20:24:25.835726 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:25.840449 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25437
I20260928 20:24:25.840538 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:25.840708 25323 external_mini_cluster.cc:1428] Setting key dacd8ce1e91120dd56fbc5df6f47fd16
Sep 28 20:24:25 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627065, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:25.847100 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:25.836508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44633 (local address 127.24.186.193:45933)
0928 20:24:25.836872 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:25.836973 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:25.837516 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:25.838133 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:25.838138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:25.838736 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:25.838940 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:25.838945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:25.839664 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:25.839666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:25.840694 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:25.840697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:25.841310 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:25.841311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:25.841463 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:25.842041 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:25.842050 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:25.843586 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:25.845671 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:25.845678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:25.845680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:25.845914 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:25.846115 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:25.846116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:25.846116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:25.846215 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:25.846530 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:25.846535 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:25.846656 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 20:24:25.846740 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:24:25.846926 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":140,"threads_started":1}
I20260928 20:24:25.847103 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:25.836792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:45933 (local address 127.24.186.254:44633)
0928 20:24:25.836959 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:25.837043 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:25.837095 (+ 52us) server_negotiation.cc:407] Connection header received
0928 20:24:25.837607 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:25.837608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:25.837621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:25.837705 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:25.839044 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:25.839561 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:25.840828 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:25.841210 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:25.841298 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:25.843705 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:25.843707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:25.843707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:25.843716 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:25.845575 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:25.845973 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:25.845974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:25.845974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:25.846043 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:25.846283 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:25.846284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:25.846285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:25.846452 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:25.846543 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:25.846837 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 20:24:25.846947 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260928 20:24:25.847693 25566 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44633
I20260928 20:24:25.847833 25566 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:25.848064 25566 heartbeater.cc:511] Master 127.24.186.254:44633 requested a full tablet report, sending...
I20260928 20:24:25.848747 25375 ts_manager.cc:194] Registered new tserver with Master: cd26575c2a3b4b9ab8f003c3fe569717 (127.24.186.193:46769)
I20260928 20:24:25.849388 25375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:45933
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
W20260928 20:24:26.630792 25323 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.24.186.194: real 0.789s user 0.000s sys 0.004s
I20260928 20:24:26.631165 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:26.718307 25572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:26.718549 25572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:26.718577 25572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:26.720114 25572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:26.720172 25572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:26.720203 25572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:26.721911 25572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:26.722133 25572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:26.722342 25572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:26.727851 25578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:26.728013 25572 server_base.cc:1034] running on GCE node
W20260928 20:24:26.727850 25580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:26.728020 25577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:26.728392 25572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:26.728659 25572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:26.729838 25572 hybrid_clock.cc:648] HybridClock initialized: now 1790627066729866 us; error 95 us; skew 500 ppm
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:26.739779 25572 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:26.740680 25572 webserver.cc:460] Webserver started at http://127.24.186.194:43891/ using document root <none> and password file <none>
I20260928 20:24:26.740907 25572 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:26.740953 25572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:26.741050 25572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:26.742129 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "46144924d96c4e6c83258077cfc276d0"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "949a9f30d40fed60067e64beb6cc6dc0"
server_key_iv: "26328a2ad1a8b1e0b72c6b13ddd9cdf6"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.742460 25572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "46144924d96c4e6c83258077cfc276d0"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "949a9f30d40fed60067e64beb6cc6dc0"
server_key_iv: "26328a2ad1a8b1e0b72c6b13ddd9cdf6"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.744017 25572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:26.745182 25587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.745744 25572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:26.745908 25572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "46144924d96c4e6c83258077cfc276d0"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "949a9f30d40fed60067e64beb6cc6dc0"
server_key_iv: "26328a2ad1a8b1e0b72c6b13ddd9cdf6"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.746131 25572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:26.770980 25572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:26.773183 25572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:26.773370 25572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:26.773617 25572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:26.773954 25572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:26.773988 25572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.774010 25572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:26.774022 25572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.781369 25572 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:40359
I20260928 20:24:26.781412 25700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:40359 every 8 connection(s)
I20260928 20:24:26.781826 25572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:26.787326 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25572
I20260928 20:24:26.787432 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:26.787611 25323 external_mini_cluster.cc:1428] Setting key beb0b51afe25c74a2c544e949ce647ea
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:26.792871 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:26.782641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:32861 (local address 127.24.186.254:44633)
0928 20:24:26.782780 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:26.782782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:26.782953 (+ 171us) server_negotiation.cc:407] Connection header received
0928 20:24:26.783522 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:26.783523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:26.783539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:26.783595 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:26.784681 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.785244 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.786300 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.786505 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.786586 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:26.789330 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:26.789332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:26.789332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:26.789341 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:26.791177 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.791628 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.791629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.791629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.791713 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.792047 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.792048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.792049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.792211 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:26.792286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:26.792642 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 20:24:26.792749 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260928 20:24:26.792888 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:26.782619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44633 (local address 127.24.186.194:32861)
0928 20:24:26.782806 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:26.782853 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:26.783401 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:26.783686 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:26.783693 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:26.784362 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:26.784566 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:26.784571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.785349 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.785350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:26.786201 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:26.786204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.786593 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.786594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:26.786741 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:26.787269 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:26.787281 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:26.789220 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:26.791278 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:26.791285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:26.791287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:26.791544 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:26.791782 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:26.791783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:26.791783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:26.791974 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:26.792286 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:26.792291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:26.792486 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 20:24:26.792573 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:24:26.792716 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260928 20:24:26.793406 25701 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44633
I20260928 20:24:26.793524 25701 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:26.793773 25701 heartbeater.cc:511] Master 127.24.186.254:44633 requested a full tablet report, sending...
I20260928 20:24:26.794235 25375 ts_manager.cc:194] Registered new tserver with Master: 46144924d96c4e6c83258077cfc276d0 (127.24.186.194:40359)
I20260928 20:24:26.794623 25375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:32861
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:26.815502 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:24:26.850639 25566 heartbeater.cc:503] Master 127.24.186.254:44633 was elected leader, sending a full tablet report...
W20260928 20:24:26.900585 25707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:26.900801 25707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:26.900830 25707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:26.902094 25707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:26.902168 25707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:26.902194 25707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:26.903715 25707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44633
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:26.903985 25707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:26.904253 25707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:26.909528 25715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:26.909528 25712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:26.909557 25713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:26.909951 25707 server_base.cc:1034] running on GCE node
I20260928 20:24:26.910164 25707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:26.910454 25707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:26.911592 25707 hybrid_clock.cc:648] HybridClock initialized: now 1790627066911586 us; error 32 us; skew 500 ppm
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:26.914280 25707 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:26.914765 25707 webserver.cc:460] Webserver started at http://127.24.186.195:36173/ using document root <none> and password file <none>
I20260928 20:24:26.914991 25707 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:26.915035 25707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:26.915175 25707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:26.916296 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "4da3b93a83854b939f5f51f299a55c64"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "197601fd542318440c15399af8c3e96e"
server_key_iv: "224616a3ecadb10b3280ceeb56988ba9"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.916656 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "4da3b93a83854b939f5f51f299a55c64"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "197601fd542318440c15399af8c3e96e"
server_key_iv: "224616a3ecadb10b3280ceeb56988ba9"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.918383 25707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:26.919240 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.919430 25707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:26.919538 25707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "4da3b93a83854b939f5f51f299a55c64"
format_stamp: "Formatted at 2026-09-28 20:24:26 on dist-test-slave-k477"
server_key: "197601fd542318440c15399af8c3e96e"
server_key_iv: "224616a3ecadb10b3280ceeb56988ba9"
server_key_version: "encryptionkey@0"
I20260928 20:24:26.919616 25707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:26.931444 25707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:26.934201 25707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:26.934615 25707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:26.934986 25707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:26.935341 25707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:26.935397 25707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.935432 25707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:26.935460 25707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:26.941550 25707 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:44027
I20260928 20:24:26.941591 25835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:44027 every 8 connection(s)
I20260928 20:24:26.941941 25707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:26.950937 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25707
I20260928 20:24:26.951058 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:26.951263 25323 external_mini_cluster.cc:1428] Setting key 335c2bd77e09326e263f13b0d2e9c344
I20260928 20:24:26.951676 25703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:26.942775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:45049 (local address 127.24.186.254:44633)
0928 20:24:26.942869 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:26.942871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:26.942966 (+ 95us) server_negotiation.cc:407] Connection header received
0928 20:24:26.943526 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:26.943527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:26.943537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:26.943588 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:26.944648 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.945281 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.946318 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.946519 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.946582 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:26.948787 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:26.948788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:26.948789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:26.948796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:26.950201 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.950586 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.950587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.950587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.950636 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.950926 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.950927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.950927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.951082 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:26.951148 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:26.951451 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 20:24:26.951563 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 20:24:26.951910 25838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:26.942689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44633 (local address 127.24.186.195:45049)
0928 20:24:26.942855 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:26.942894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:26.943430 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:26.943642 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:26.943645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:26.944345 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:26.944539 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:26.944545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.945384 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.945385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:26.946208 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:26.946227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.946599 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.946600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:26.946704 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:26.947206 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:26.947228 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:26.948687 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:26.950279 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:26.950285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:26.950288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:26.950519 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:26.950711 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:26.950712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:26.950712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:26.950831 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:26.951138 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:26.951144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:26.951287 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 20:24:26.951418 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 20:24:26.951586 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260928 20:24:26.952363 25836 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44633
I20260928 20:24:26.952469 25836 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:26.952651 25836 heartbeater.cc:511] Master 127.24.186.254:44633 requested a full tablet report, sending...
I20260928 20:24:26.953205 25375 ts_manager.cc:194] Registered new tserver with Master: 4da3b93a83854b939f5f51f299a55c64 (127.24.186.195:44027)
I20260928 20:24:26.953665 25375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:45049
I20260928 20:24:26.963642 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:26.989225 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:26.980782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50798 (local address 127.24.186.193:46769)
0928 20:24:26.980962 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:26.980967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:26.980979 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:26.981006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:26.981008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:26.981022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:26.981091 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:26.981663 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.982398 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.983185 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:26.983531 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:26.983590 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:26.986119 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:26.986142 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:26.986142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:26.986150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:26.987926 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.988326 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.988330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.988330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.988399 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:26.988610 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:26.988611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:26.988612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:26.988778 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:26.988948 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:26.989008 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 20:24:26.989089 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260928 20:24:26.989595 25480 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50798
Sep 28 20:24:26 dist-test-slave-k477 krb5kdc[25330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:24:27 dist-test-slave-k477 krb5kdc[25330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:27.014263 25843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:27.006505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.24.186.193:46769)
0928 20:24:27.006572 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:27.006574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:27.006587 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:24:27.006703 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:27.006704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:27.006718 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:27.006778 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:27.007363 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:27.007939 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:27.008688 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:27.008891 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:27.008944 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:27.011450 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:27.011451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:27.011452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:27.011458 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:27.013187 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:27.013554 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:27.013555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:27.013555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:27.013614 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:27.013802 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:27.013802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:27.013802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:27.013901 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:27.014004 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:27.014127 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 20:24:27.014192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:24:27.014806 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25437
I20260928 20:24:27.019749 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25572
I20260928 20:24:27.025605 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25707
I20260928 20:24:27.030198 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25343
2026-09-28T20:24:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6107 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:27 dist-test-slave-k477 krb5kdc[25850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:27 dist-test-slave-k477 krb5kdc[25850](info): set up 2 sockets
Sep 28 20:24:27 dist-test-slave-k477 krb5kdc[25850](info): commencing operation
krb5kdc: starting...
W20260928 20:24:28.088109 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:28.227382 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:42739
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:42739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:28.312230 25863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:28.312511 25863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:28.312553 25863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:28.313823 25863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:28.313907 25863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:28.313935 25863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:28.313968 25863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:28.313993 25863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:28.315618 25863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:42739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:42739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:28.316133 25863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:28.316421 25863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:28.321839 25871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:28.321882 25869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.321921 25863 server_base.cc:1034] running on GCE node
W20260928 20:24:28.321864 25868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.322360 25863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:28.322679 25863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:28.323904 25863 hybrid_clock.cc:648] HybridClock initialized: now 1790627068323871 us; error 50 us; skew 500 ppm
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:28.326622 25863 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:28.327135 25863 webserver.cc:460] Webserver started at http://127.24.186.254:43253/ using document root <none> and password file <none>
I20260928 20:24:28.327404 25863 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:28.327450 25863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:28.327622 25863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:28.328775 25863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "91fa7003b34347b394da1b6a8684c146"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "6723dd56581a8f6711e8a4c58e35dc0c"
server_key_iv: "b21c82e3ead4ad9a77ce56a5a9a4da40"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.329144 25863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "91fa7003b34347b394da1b6a8684c146"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "6723dd56581a8f6711e8a4c58e35dc0c"
server_key_iv: "b21c82e3ead4ad9a77ce56a5a9a4da40"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.330616 25863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:28.331485 25878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.331703 25863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:28.331843 25863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "91fa7003b34347b394da1b6a8684c146"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "6723dd56581a8f6711e8a4c58e35dc0c"
server_key_iv: "b21c82e3ead4ad9a77ce56a5a9a4da40"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.331943 25863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:28.353224 25863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:28.355166 25863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:28.355342 25863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:28.359773 25863 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:42739
I20260928 20:24:28.359839 25930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:42739 every 8 connection(s)
I20260928 20:24:28.360257 25863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:28.361141 25931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:28.362582 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25863
I20260928 20:24:28.362674 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:28.362835 25323 external_mini_cluster.cc:1428] Setting key 4d09f77c7230a54d3bc28eefa41ff626
I20260928 20:24:28.364168 25931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: Bootstrap starting.
I20260928 20:24:28.364862 25931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:28.365200 25931 log.cc:824] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:28.365991 25931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: No bootstrap required, opened a new log
I20260928 20:24:28.368863 25931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } }
I20260928 20:24:28.369081 25931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:28.369120 25931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91fa7003b34347b394da1b6a8684c146, State: Initialized, Role: FOLLOWER
I20260928 20:24:28.369295 25931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } }
I20260928 20:24:28.369423 25931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:28.369477 25931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:28.369537 25931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:28.370172 25931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } }
I20260928 20:24:28.370313 25931 leader_election.cc:302] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91fa7003b34347b394da1b6a8684c146; no voters:
I20260928 20:24:28.370512 25931 leader_election.cc:288] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:28.370579 25936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:28.370800 25936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [term 1 LEADER]: Becoming Leader. State: Replica: 91fa7003b34347b394da1b6a8684c146, State: Running, Role: LEADER
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:28.370841 25931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:28.370962 25936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } }
I20260928 20:24:28.372493 25938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91fa7003b34347b394da1b6a8684c146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } } }
I20260928 20:24:28.372525 25937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91fa7003b34347b394da1b6a8684c146. Latest consensus state: current_term: 1 leader_uuid: "91fa7003b34347b394da1b6a8684c146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91fa7003b34347b394da1b6a8684c146" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42739 } } }
I20260928 20:24:28.372931 25937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:28.373034 25951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 20:24:28.372929 25938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146 [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:28.373147 25951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:28.375545 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.363461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.24.186.254:42739)
0928 20:24:28.363660 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.363672 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.363708 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:24:28.364303 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.364312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.364372 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.364536 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:28.365187 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.366552 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.367744 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.368537 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.368686 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.371581 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:28.371586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:28.371586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:28.371610 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:28.373628 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.374024 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.374028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.374028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.374128 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.374345 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.374346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.374346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.374614 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:28.374781 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.374901 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.375238 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260928 20:24:28.376003 25953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:28.376349 25953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:28.378129 25953 catalog_manager.cc:1347] Generated new cluster ID: 7cff0bcfaa33489f9d74d9aaa5ae74c1
I20260928 20:24:28.378187 25953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:28.393729 25953 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:28.394436 25953 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:28.400707 25953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91fa7003b34347b394da1b6a8684c146: Generated new TSK 0
I20260928 20:24:28.400947 25953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:28.431576 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:28.517110 25957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:28.517333 25957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:28.517367 25957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:28.518760 25957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:28.518816 25957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:28.518847 25957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:28.520434 25957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:28.520658 25957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:28.520844 25957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:28.526669 25963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:28.526726 25965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:28.526669 25962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.527000 25957 server_base.cc:1034] running on GCE node
I20260928 20:24:28.527268 25957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:28.527537 25957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:28.528692 25957 hybrid_clock.cc:648] HybridClock initialized: now 1790627068528663 us; error 39 us; skew 500 ppm
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:28.531271 25957 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:28.531749 25957 webserver.cc:460] Webserver started at http://127.24.186.193:41513/ using document root <none> and password file <none>
I20260928 20:24:28.532016 25957 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:28.532065 25957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:28.532166 25957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:28.533175 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "354c09deb0ca4b8babd69b602b5e3a39"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "45f19ee5bb0eca2b0876fa5a2b0c39df"
server_key_iv: "66a4df810477f3379930b80822072a39"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.533545 25957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "354c09deb0ca4b8babd69b602b5e3a39"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "45f19ee5bb0eca2b0876fa5a2b0c39df"
server_key_iv: "66a4df810477f3379930b80822072a39"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.535036 25957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:28.535954 25972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.536108 25957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:28.536201 25957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "354c09deb0ca4b8babd69b602b5e3a39"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "45f19ee5bb0eca2b0876fa5a2b0c39df"
server_key_iv: "66a4df810477f3379930b80822072a39"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.536262 25957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:28.553795 25957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:28.556295 25957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:28.556420 25957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:28.556609 25957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:28.556955 25957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:28.556993 25957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.557016 25957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:28.557063 25957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.564199 25957 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:45249
I20260928 20:24:28.564373 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:45249 every 8 connection(s)
I20260928 20:24:28.564783 25957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:28.567718 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 25957
I20260928 20:24:28.567885 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:28.568075 25323 external_mini_cluster.cc:1428] Setting key 6fdbb4cf9124e001225cd070012613f5
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:24:28.576484 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.565914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36819 (local address 127.24.186.254:42739)
0928 20:24:28.566017 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.566020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.566236 (+ 216us) server_negotiation.cc:407] Connection header received
0928 20:24:28.567240 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.567241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.567253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.567315 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:28.568217 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.568933 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.570027 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.570248 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.570375 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.573270 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:28.573271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:28.573272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:28.573280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:28.574787 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.575278 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.575279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.575280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.575343 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.575617 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.575618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.575619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.575757 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:28.575968 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.576272 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.576377 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 20:24:28.576695 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.565789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42739 (local address 127.24.186.193:36819)
0928 20:24:28.566062 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.566109 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.567095 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.567421 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.567425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.567721 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:28.568070 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.568076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.569037 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.569037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.569946 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.569950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.570384 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.570385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.570530 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.571166 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:28.571176 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:28.573137 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:28.574912 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.574922 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.574925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.575194 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.575416 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.575417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.575418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.575545 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.575931 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:28.575939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:28.576118 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.576247 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.576432 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":190,"threads_started":1}
I20260928 20:24:28.577245 26086 heartbeater.cc:347] Connected to a master server at 127.24.186.254:42739
I20260928 20:24:28.577358 26086 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:28.577554 26086 heartbeater.cc:511] Master 127.24.186.254:42739 requested a full tablet report, sending...
I20260928 20:24:28.578126 25895 ts_manager.cc:194] Registered new tserver with Master: 354c09deb0ca4b8babd69b602b5e3a39 (127.24.186.193:45249)
I20260928 20:24:28.578859 25895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:36819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:28.595683 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:28.684103 26091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:28.684338 26091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:28.684366 26091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:28.685518 26091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:28.685561 26091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:28.685590 26091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:28.687358 26091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:28.687651 26091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:28.687964 26091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:28.693073 26097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.693233 26091 server_base.cc:1034] running on GCE node
W20260928 20:24:28.693094 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:28.693082 26096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.693594 26091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:28.693857 26091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:28.695075 26091 hybrid_clock.cc:648] HybridClock initialized: now 1790627068695070 us; error 49 us; skew 500 ppm
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:28.697757 26091 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:28.698240 26091 webserver.cc:460] Webserver started at http://127.24.186.194:41355/ using document root <none> and password file <none>
I20260928 20:24:28.698493 26091 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:28.698563 26091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:28.698719 26091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:28.700068 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "bb704d5a0ede45ed97063a5ecd06d8d0"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "edf02ea8e7a666125368974545fca81a"
server_key_iv: "b7abc9b8415da44cea21d7c39cb86216"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.700472 26091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "bb704d5a0ede45ed97063a5ecd06d8d0"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "edf02ea8e7a666125368974545fca81a"
server_key_iv: "b7abc9b8415da44cea21d7c39cb86216"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.702013 26091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:28.702868 26106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.703054 26091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:28.703173 26091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "bb704d5a0ede45ed97063a5ecd06d8d0"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "edf02ea8e7a666125368974545fca81a"
server_key_iv: "b7abc9b8415da44cea21d7c39cb86216"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.703254 26091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:28.718200 26091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:28.720367 26091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:28.720544 26091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:28.720788 26091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:28.721105 26091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:28.721167 26091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.721210 26091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:28.721238 26091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.727248 26091 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:39235
I20260928 20:24:28.727291 26219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:39235 every 8 connection(s)
I20260928 20:24:28.727636 26091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:28.731364 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26091
I20260928 20:24:28.731498 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:28.731703 25323 external_mini_cluster.cc:1428] Setting key c7da0482cd8c4c387942bd6f6fd68230
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:28.740805 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.728609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40175 (local address 127.24.186.254:42739)
0928 20:24:28.728698 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.728701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.729087 (+ 386us) server_negotiation.cc:407] Connection header received
0928 20:24:28.730082 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.730083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.730095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.730154 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:28.730957 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.731871 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.733723 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.734081 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.734175 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.737210 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:28.737212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:28.737212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:28.737220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:28.739290 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.739734 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.739735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.739736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.739838 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.740089 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.740090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.740090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.740241 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:28.740322 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.740606 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.740708 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 20:24:28.741081 26222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.728596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42739 (local address 127.24.186.194:40175)
0928 20:24:28.728810 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.728969 (+ 159us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.729978 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.730227 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.730231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.730506 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:28.730720 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.730724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.732083 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.732085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.733501 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.733508 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.734181 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.734182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.734354 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.735034 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:28.735046 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:28.737050 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:28.739378 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.739388 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.739391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.739643 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.739902 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.739905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.739905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.740016 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.740312 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:28.740318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:28.740466 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.740585 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.740757 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
I20260928 20:24:28.741585 26220 heartbeater.cc:347] Connected to a master server at 127.24.186.254:42739
I20260928 20:24:28.741710 26220 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:28.741945 26220 heartbeater.cc:511] Master 127.24.186.254:42739 requested a full tablet report, sending...
I20260928 20:24:28.742463 25895 ts_manager.cc:194] Registered new tserver with Master: bb704d5a0ede45ed97063a5ecd06d8d0 (127.24.186.194:39235)
I20260928 20:24:28.742873 25895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:40175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:28.763615 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:28.856739 26225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:28.856972 26225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:28.857002 26225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:28.858327 26225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:28.858382 26225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:28.858412 26225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:28.860102 26225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:42739
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:28.860376 26225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:28.860631 26225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:28.865952 26230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:28.865980 26231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.865998 26225 server_base.cc:1034] running on GCE node
W20260928 20:24:28.865968 26233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:28.866382 26225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:28.866660 26225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:28.867893 26225 hybrid_clock.cc:648] HybridClock initialized: now 1790627068867866 us; error 28 us; skew 500 ppm
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:28.870734 26225 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:28.871333 26225 webserver.cc:460] Webserver started at http://127.24.186.195:45341/ using document root <none> and password file <none>
I20260928 20:24:28.871585 26225 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:28.871654 26225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:28.871793 26225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:28.873679 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "0b44e55cec894b8bbdf98df9ba288a67"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "63a1552cb5cd6cff533ee2041067a156"
server_key_iv: "06f983e47a6e5d1a5df6bd28a3929f29"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.874087 26225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "0b44e55cec894b8bbdf98df9ba288a67"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "63a1552cb5cd6cff533ee2041067a156"
server_key_iv: "06f983e47a6e5d1a5df6bd28a3929f29"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.875728 26225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:28.876688 26240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.877071 26225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:28.877192 26225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "0b44e55cec894b8bbdf98df9ba288a67"
format_stamp: "Formatted at 2026-09-28 20:24:28 on dist-test-slave-k477"
server_key: "63a1552cb5cd6cff533ee2041067a156"
server_key_iv: "06f983e47a6e5d1a5df6bd28a3929f29"
server_key_version: "encryptionkey@0"
I20260928 20:24:28.877296 26225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:28.888288 26225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:28.890743 26225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:28.890920 26225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:28.891184 26225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:28.891508 26225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:28.891562 26225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.891604 26225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:28.891629 26225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:28.897809 26225 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:44989
I20260928 20:24:28.897933 26353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:44989 every 8 connection(s)
I20260928 20:24:28.898329 26225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:28.900082 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26225
I20260928 20:24:28.900213 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:28.900416 25323 external_mini_cluster.cc:1428] Setting key 498b7f069fe746d57914c82e3a4d8b7c
Sep 28 20:24:28 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:28.909207 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.899259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:38791 (local address 127.24.186.254:42739)
0928 20:24:28.899330 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.899333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.899522 (+ 189us) server_negotiation.cc:407] Connection header received
0928 20:24:28.900882 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.900883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.900895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.900941 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:28.901699 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.902232 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.903289 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.903494 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.903574 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.906034 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:28.906036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:28.906036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:28.906045 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:28.907661 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.908162 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.908163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.908164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.908235 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.908500 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.908501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.908501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.908639 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:28.908721 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.909019 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.909107 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:24:28.909431 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.899195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42739 (local address 127.24.186.195:38791)
0928 20:24:28.899389 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.899428 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.900788 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.901014 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.901018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.901401 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:28.901599 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.901604 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.902340 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.902341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.903164 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.903167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.903593 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.903593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.903732 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.904381 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:28.904389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:28.905921 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:28.907756 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.907765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.907768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.908057 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.908304 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:28.908305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:28.908306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:28.908425 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:28.908710 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:28.908717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:28.908875 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.908987 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.909171 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 20:24:28.909920 26354 heartbeater.cc:347] Connected to a master server at 127.24.186.254:42739
I20260928 20:24:28.910043 26354 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:28.910235 26354 heartbeater.cc:511] Master 127.24.186.254:42739 requested a full tablet report, sending...
I20260928 20:24:28.910646 25895 ts_manager.cc:194] Registered new tserver with Master: 0b44e55cec894b8bbdf98df9ba288a67 (127.24.186.195:44989)
I20260928 20:24:28.910990 25895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:38791
I20260928 20:24:28.912288 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:24:28.920462 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.914010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.24.186.254:42739)
0928 20:24:28.914178 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.914181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.914301 (+ 120us) server_negotiation.cc:407] Connection header received
0928 20:24:28.914384 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.914385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.914396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.914454 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:28.915032 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.915529 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.916320 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.916524 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.916612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.917865 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:28.917867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:28.917867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:28.917876 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:28.919203 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.919580 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.919581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.919581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.919639 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:28.919865 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:28.919866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:28.919867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:28.920030 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:28.920144 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.920270 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.920355 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 20:24:28.922792 25895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:28.924458 25895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:28.931591 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.24.186.194:39235)
0928 20:24:28.927484 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.927487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.927495 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:28.927527 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.927529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.927541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.927635 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:28.927881 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929018 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.930448 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931138 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931217 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931276 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.931446 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.931497 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260928 20:24:28.931583 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39235 (local address 127.0.0.1:33338)
0928 20:24:28.927204 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.927248 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.927319 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.927645 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.927650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.927653 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:28.927734 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.927736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929129 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.929130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.930355 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.930357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931244 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.931326 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931339 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.931414 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.931478 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":80,"thread_start_us":108,"threads_started":1}
I20260928 20:24:28.931892 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45249 (local address 127.0.0.1:40014)
0928 20:24:28.927464 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.927475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.927528 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.927955 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.927956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.927958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:28.928002 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.928004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929254 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.929255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.930615 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.930618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931576 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.931660 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931667 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.931722 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.931764 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":41,"thread_start_us":41,"threads_started":1}
I20260928 20:24:28.932120 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.24.186.193:45249)
0928 20:24:28.927767 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.927771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.927781 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:28.927837 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.927839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.927855 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.928071 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:28.928158 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929169 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.930706 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931425 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931494 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931541 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.931998 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.932050 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":94,"threads_started":1}
I20260928 20:24:28.932597 26154 tablet_service.cc:1467] Processing CreateTablet for tablet f9e0c8eec0ef49628ba45ed157f3153a (DEFAULT_TABLE table=test-table [id=1c5dcb8ca91b46e781203568c4e23b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:28.932682 26020 tablet_service.cc:1467] Processing CreateTablet for tablet f9e0c8eec0ef49628ba45ed157f3153a (DEFAULT_TABLE table=test-table [id=1c5dcb8ca91b46e781203568c4e23b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:28.932968 26154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e0c8eec0ef49628ba45ed157f3153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:28.933045 26020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e0c8eec0ef49628ba45ed157f3153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:28.935958 26371 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Bootstrap starting.
I20260928 20:24:28.935985 26372 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Bootstrap starting.
I20260928 20:24:28.936197 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.24.186.195:44989)
0928 20:24:28.927716 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:28.927721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:28.927731 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:28.927770 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:28.927773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:28.927788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:28.927886 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:28.928117 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929048 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.930725 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931241 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931310 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931364 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:28.931620 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 20:24:28.931668 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":109,"threads_started":1}
I20260928 20:24:28.931586 26365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:28.927019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44989 (local address 127.0.0.1:33160)
0928 20:24:28.927545 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:28.927555 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:28.927599 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:28.927972 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:28.927973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:28.927975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:28.928036 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.928037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.929217 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.929219 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.930616 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:28.930618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:28.931397 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:28.931397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:28.931450 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:28.931453 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:28.931496 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 20:24:28.931527 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":100,"thread_start_us":102,"threads_started":1}
I20260928 20:24:28.936751 26372 tablet_bootstrap.cc:654] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:28.937099 26371 tablet_bootstrap.cc:654] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:28.937182 26288 tablet_service.cc:1467] Processing CreateTablet for tablet f9e0c8eec0ef49628ba45ed157f3153a (DEFAULT_TABLE table=test-table [id=1c5dcb8ca91b46e781203568c4e23b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:28.937376 26372 log.cc:824] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:28.937587 26288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e0c8eec0ef49628ba45ed157f3153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:28.937861 26371 log.cc:824] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:28.938436 26372 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: No bootstrap required, opened a new log
I20260928 20:24:28.938625 26372 ts_tablet_manager.cc:1397] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:24:28.938957 26371 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: No bootstrap required, opened a new log
I20260928 20:24:28.939090 26371 ts_tablet_manager.cc:1397] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:24:28.940802 26372 raft_consensus.cc:348] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.941017 26372 raft_consensus.cc:374] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:28.941063 26372 raft_consensus.cc:729] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354c09deb0ca4b8babd69b602b5e3a39, State: Initialized, Role: FOLLOWER
I20260928 20:24:28.941196 26372 consensus_queue.cc:260] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.941547 26086 heartbeater.cc:503] Master 127.24.186.254:42739 was elected leader, sending a full tablet report...
I20260928 20:24:28.941797 26372 ts_tablet_manager.cc:1428] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:28.941900 26376 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: Bootstrap starting.
I20260928 20:24:28.941690 26371 raft_consensus.cc:348] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.942205 26371 raft_consensus.cc:374] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:28.942288 26371 raft_consensus.cc:729] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb704d5a0ede45ed97063a5ecd06d8d0, State: Initialized, Role: FOLLOWER
I20260928 20:24:28.942476 26371 consensus_queue.cc:260] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.942643 26376 tablet_bootstrap.cc:654] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:28.942834 26371 ts_tablet_manager.cc:1428] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 20:24:28.943038 26220 heartbeater.cc:503] Master 127.24.186.254:42739 was elected leader, sending a full tablet report...
I20260928 20:24:28.943241 26376 log.cc:824] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:28.944180 26376 tablet_bootstrap.cc:492] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: No bootstrap required, opened a new log
I20260928 20:24:28.944244 26376 ts_tablet_manager.cc:1397] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:28.945672 26376 raft_consensus.cc:348] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.945824 26376 raft_consensus.cc:374] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:28.945847 26376 raft_consensus.cc:729] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b44e55cec894b8bbdf98df9ba288a67, State: Initialized, Role: FOLLOWER
I20260928 20:24:28.945998 26376 consensus_queue.cc:260] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:28.946362 26376 ts_tablet_manager.cc:1428] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:28.946363 26354 heartbeater.cc:503] Master 127.24.186.254:42739 was elected leader, sending a full tablet report...
W20260928 20:24:28.978423 26221 tablet.cc:2369] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:29.001693 26375 raft_consensus.cc:482] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:29.001837 26375 raft_consensus.cc:504] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:29.002388 26375 leader_election.cc:288] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb704d5a0ede45ed97063a5ecd06d8d0 (127.24.186.194:39235), 0b44e55cec894b8bbdf98df9ba288a67 (127.24.186.195:44989)
I20260928 20:24:29.005733 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.002739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:41773 (local address 127.24.186.195:44989)
0928 20:24:29.002793 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:29.002795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:29.003001 (+ 206us) server_negotiation.cc:407] Connection header received
0928 20:24:29.003035 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:29.003036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:29.003046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:29.003104 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:29.003250 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.003711 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.004947 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.005350 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.005412 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.005419 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:29.005601 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:24:29.005649 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:24:29.005844 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.002739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36331 (local address 127.24.186.194:39235)
0928 20:24:29.002788 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:29.002791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:29.002803 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:29.002837 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:29.002838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:29.002851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:29.002919 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:29.003106 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.003648 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.004947 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.005369 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.005469 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.005478 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:29.005714 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 20:24:29.005761 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:24:29.005820 26088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.002624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39235 (local address 127.24.186.193:36331)
0928 20:24:29.002695 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:29.002709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:29.002779 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:29.002923 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:29.002924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:29.002926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:29.003018 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:29.003022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.003744 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.003745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:29.004848 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:29.004851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.005520 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.005521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:29.005617 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.005621 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:29.005676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:24:29.005715 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":32}
I20260928 20:24:29.005816 26380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.002626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44989 (local address 127.24.186.193:41773)
0928 20:24:29.002912 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:29.002923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:29.002991 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:29.003107 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:29.003108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:29.003109 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:29.003175 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:29.003177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.003831 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.003831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:29.004847 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:29.004850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.005463 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.005465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:29.005547 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.005551 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:29.005594 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 20:24:29.005635 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":86,"thread_start_us":119,"threads_started":1}
I20260928 20:24:29.006237 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e0c8eec0ef49628ba45ed157f3153a" candidate_uuid: "354c09deb0ca4b8babd69b602b5e3a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b44e55cec894b8bbdf98df9ba288a67" is_pre_election: true
I20260928 20:24:29.006230 26174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e0c8eec0ef49628ba45ed157f3153a" candidate_uuid: "354c09deb0ca4b8babd69b602b5e3a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" is_pre_election: true
I20260928 20:24:29.006395 26308 raft_consensus.cc:2454] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354c09deb0ca4b8babd69b602b5e3a39 in term 0.
I20260928 20:24:29.006394 26174 raft_consensus.cc:2454] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354c09deb0ca4b8babd69b602b5e3a39 in term 0.
I20260928 20:24:29.006655 25976 leader_election.cc:302] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b44e55cec894b8bbdf98df9ba288a67, 354c09deb0ca4b8babd69b602b5e3a39; no voters:
I20260928 20:24:29.006805 26375 raft_consensus.cc:2781] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:29.006865 26375 raft_consensus.cc:482] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:29.006888 26375 raft_consensus.cc:3037] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:29.007586 26375 raft_consensus.cc:504] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:29.007738 26375 leader_election.cc:288] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [CANDIDATE]: Term 1 election: Requested vote from peers bb704d5a0ede45ed97063a5ecd06d8d0 (127.24.186.194:39235), 0b44e55cec894b8bbdf98df9ba288a67 (127.24.186.195:44989)
I20260928 20:24:29.007942 26174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e0c8eec0ef49628ba45ed157f3153a" candidate_uuid: "354c09deb0ca4b8babd69b602b5e3a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0"
I20260928 20:24:29.008028 26174 raft_consensus.cc:3037] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:29.008683 26174 raft_consensus.cc:2454] T f9e0c8eec0ef49628ba45ed157f3153a P bb704d5a0ede45ed97063a5ecd06d8d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354c09deb0ca4b8babd69b602b5e3a39 in term 1.
I20260928 20:24:29.008810 26308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e0c8eec0ef49628ba45ed157f3153a" candidate_uuid: "354c09deb0ca4b8babd69b602b5e3a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b44e55cec894b8bbdf98df9ba288a67"
I20260928 20:24:29.008898 26308 raft_consensus.cc:3037] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:29.008970 25973 leader_election.cc:302] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354c09deb0ca4b8babd69b602b5e3a39, bb704d5a0ede45ed97063a5ecd06d8d0; no voters:
I20260928 20:24:29.009099 26375 raft_consensus.cc:2781] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:29.009307 26375 raft_consensus.cc:686] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [term 1 LEADER]: Becoming Leader. State: Replica: 354c09deb0ca4b8babd69b602b5e3a39, State: Running, Role: LEADER
I20260928 20:24:29.009413 26375 consensus_queue.cc:237] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } }
I20260928 20:24:29.009789 26308 raft_consensus.cc:2454] T f9e0c8eec0ef49628ba45ed157f3153a P 0b44e55cec894b8bbdf98df9ba288a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354c09deb0ca4b8babd69b602b5e3a39 in term 1.
I20260928 20:24:29.010892 25895 catalog_manager.cc:5495] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 354c09deb0ca4b8babd69b602b5e3a39 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "354c09deb0ca4b8babd69b602b5e3a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354c09deb0ca4b8babd69b602b5e3a39" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb704d5a0ede45ed97063a5ecd06d8d0" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b44e55cec894b8bbdf98df9ba288a67" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44989 } health_report { overall_health: UNKNOWN } } }
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:29.033392 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.024329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.24.186.193:45249)
0928 20:24:29.024385 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:29.024386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:29.024535 (+ 149us) server_negotiation.cc:407] Connection header received
0928 20:24:29.024569 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:29.024570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:29.024644 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:29.024689 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:29.025518 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.026020 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.026983 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.027194 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.027271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.030031 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:29.030044 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:29.030044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:29.030060 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:29.031924 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.032297 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.032300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.032300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.032373 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.032589 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.032589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.032589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.032744 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:29.032878 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:29.033125 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 20:24:29.033287 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:29.058801 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.051097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.24.186.193:45249)
0928 20:24:29.051166 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:29.051168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:29.051229 (+ 61us) server_negotiation.cc:407] Connection header received
0928 20:24:29.051276 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:29.051276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:29.051287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:29.051339 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:29.052045 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.052557 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.053323 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.053524 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.053597 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.056172 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:29.056174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:29.056174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:29.056182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:29.057658 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.058008 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.058009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.058010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.058067 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.058284 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.058285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.058285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.058392 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:29.058475 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:29.058654 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 20:24:29.058723 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 20:24:29.065789 26087 tablet.cc:2369] T f9e0c8eec0ef49628ba45ed157f3153a P 354c09deb0ca4b8babd69b602b5e3a39: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[25850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[25850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:29.083093 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:29.075506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.24.186.193:45249)
0928 20:24:29.075587 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:29.075589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:29.075671 (+ 82us) server_negotiation.cc:407] Connection header received
0928 20:24:29.075778 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:29.075779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:29.075790 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:29.075865 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:29.076440 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.076922 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.077689 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:29.077884 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:29.077963 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:29.080473 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:29.080475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:29.080476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:29.080484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:29.081911 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.082307 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.082308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.082309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.082368 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:29.082569 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:29.082570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:29.082571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:29.082680 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:29.082810 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:29.082903 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 20:24:29.082985 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 20:24:29.083573 26000 tablet_service.cc:2908] Rejecting scan request for tablet f9e0c8eec0ef49628ba45ed157f3153a: Uninitialized: clean time has not yet been initialized
I20260928 20:24:29.084456 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25957
I20260928 20:24:29.090282 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26091
I20260928 20:24:29.095060 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26225
I20260928 20:24:29.100270 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 25863
2026-09-28T20:24:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2070 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[26395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[26395](info): set up 2 sockets
Sep 28 20:24:29 dist-test-slave-k477 krb5kdc[26395](info): commencing operation
krb5kdc: starting...
W20260928 20:24:30.164727 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:30.300062 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40279
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:30.391758 26408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:30.392084 26408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:30.392140 26408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:30.393468 26408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:30.393549 26408 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:24:30.393570 26408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:30.393594 26408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:30.393623 26408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:30.393644 26408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:30.395291 26408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:40279
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:30.395603 26408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:30.395885 26408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:30.401605 26414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:30.401871 26413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.401865 26408 server_base.cc:1034] running on GCE node
W20260928 20:24:30.402493 26416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.402772 26408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:30.403103 26408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:30.404351 26408 hybrid_clock.cc:648] HybridClock initialized: now 1790627070404335 us; error 51 us; skew 500 ppm
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:30.407620 26408 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:30.408149 26408 webserver.cc:460] Webserver started at http://127.24.186.254:41485/ using document root <none> and password file <none>
I20260928 20:24:30.408360 26408 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:30.408403 26408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:30.408499 26408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:30.409821 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "8d741867978e4d0398d853dcf3f2d53d"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "19ef8b1341682d70a0f92e8587ac8528"
server_key_iv: "99e9c0aa4234228205fbc889e69d7ccb"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.410180 26408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "8d741867978e4d0398d853dcf3f2d53d"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "19ef8b1341682d70a0f92e8587ac8528"
server_key_iv: "99e9c0aa4234228205fbc889e69d7ccb"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.412173 26408 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:24:30.412978 26423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.413188 26408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:30.413285 26408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "8d741867978e4d0398d853dcf3f2d53d"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "19ef8b1341682d70a0f92e8587ac8528"
server_key_iv: "99e9c0aa4234228205fbc889e69d7ccb"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.413352 26408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:30.437736 26408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:30.440552 26408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:30.440682 26408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:30.445504 26408 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:40279
I20260928 20:24:30.445602 26485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:40279 every 8 connection(s)
I20260928 20:24:30.446096 26408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:30.446820 26486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:30.447227 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26408
I20260928 20:24:30.447326 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:30.447487 25323 external_mini_cluster.cc:1428] Setting key 33c5a1396b42075a8ad304afad86af02
I20260928 20:24:30.449709 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Bootstrap starting.
I20260928 20:24:30.450361 26486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:30.450654 26486 log.cc:824] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:30.451390 26486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: No bootstrap required, opened a new log
I20260928 20:24:30.453339 26486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } }
I20260928 20:24:30.453531 26486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:30.453569 26486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d741867978e4d0398d853dcf3f2d53d, State: Initialized, Role: FOLLOWER
I20260928 20:24:30.453719 26486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } }
I20260928 20:24:30.453801 26486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:30.453848 26486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:30.453891 26486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:30.454720 26486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } }
I20260928 20:24:30.454824 26486 leader_election.cc:302] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d741867978e4d0398d853dcf3f2d53d; no voters:
I20260928 20:24:30.455065 26486 leader_election.cc:288] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:30.455093 26491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:30.455258 26491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [term 1 LEADER]: Becoming Leader. State: Replica: 8d741867978e4d0398d853dcf3f2d53d, State: Running, Role: LEADER
I20260928 20:24:30.455415 26486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:30.455399 26491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } }
I20260928 20:24:30.456063 26493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d741867978e4d0398d853dcf3f2d53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } } }
I20260928 20:24:30.456144 26493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:30.456293 26492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d741867978e4d0398d853dcf3f2d53d. Latest consensus state: current_term: 1 leader_uuid: "8d741867978e4d0398d853dcf3f2d53d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d741867978e4d0398d853dcf3f2d53d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40279 } } }
I20260928 20:24:30.456442 26492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:30.456977 26497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:30.457500 26497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:30.460103 26497 catalog_manager.cc:1347] Generated new cluster ID: 25feddb16fb94077aad1f79004331824
I20260928 20:24:30.460176 26497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:30.460439 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.448072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.24.186.254:40279)
0928 20:24:30.448285 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:30.448294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:30.448320 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:24:30.448736 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:30.448742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:30.448795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:30.448954 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:30.449582 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.450614 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.451383 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.451921 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.452043 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.454731 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:30.454736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:30.454736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:30.454749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:30.457191 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.457692 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.457696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.457696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.457798 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.458633 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.458636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.458637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.459231 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:30.459407 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:30.460091 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 20:24:30.460247 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260928 20:24:30.467531 26497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:30.468241 26497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:30.473073 26497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Generated new TSK 0
I20260928 20:24:30.473292 26497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:30.510737 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40279
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:30.599365 26512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:30.599615 26512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:30.599658 26512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:30.600960 26512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:30.601032 26512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:30.601070 26512 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:30.601110 26512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:30.602643 26512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:30.602929 26512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:30.603181 26512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:30.608603 26517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:30.608588 26520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:30.608608 26518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.609012 26512 server_base.cc:1034] running on GCE node
I20260928 20:24:30.609279 26512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:30.609558 26512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:30.610754 26512 hybrid_clock.cc:648] HybridClock initialized: now 1790627070610728 us; error 38 us; skew 500 ppm
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:30.613515 26512 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:30.614074 26512 webserver.cc:460] Webserver started at http://127.24.186.193:45975/ using document root <none> and password file <none>
I20260928 20:24:30.614332 26512 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:30.614405 26512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:30.614559 26512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:30.615681 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "fd595e996fc44805ab40dffe424e90bc"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "6e8b223f25508c32b2d892b2a2ed7dd2"
server_key_iv: "2a2c9e589f59cd37e4d9c5bd09e5a313"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.616492 26512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "fd595e996fc44805ab40dffe424e90bc"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "6e8b223f25508c32b2d892b2a2ed7dd2"
server_key_iv: "2a2c9e589f59cd37e4d9c5bd09e5a313"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.618165 26512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:24:30.618928 26527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.619299 26512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:30.619402 26512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "fd595e996fc44805ab40dffe424e90bc"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "6e8b223f25508c32b2d892b2a2ed7dd2"
server_key_iv: "2a2c9e589f59cd37e4d9c5bd09e5a313"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.619479 26512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:30.647214 26512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:30.649673 26512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:30.649852 26512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:30.650522 26512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:30.650583 26512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.650619 26512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:30.650638 26512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:30.660090 26512 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:38133
I20260928 20:24:30.660183 26642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:38133 every 8 connection(s)
I20260928 20:24:30.660483 26512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:30.661522 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.651135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:44873 (local address 127.24.186.254:40279)
0928 20:24:30.651216 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:30.651219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:30.651490 (+ 271us) server_negotiation.cc:407] Connection header received
0928 20:24:30.652283 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:30.652284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:30.652298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:30.652386 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:30.653017 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.653555 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.654593 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.654810 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.654897 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.658232 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:30.658234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:30.658234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:30.658243 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:30.659963 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.660457 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.660459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.660459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.660525 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.660789 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.660790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.660790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.660922 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:30.661013 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:30.661341 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 20:24:30.661435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:24:30.661707 26536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.651108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.193:44873)
0928 20:24:30.651335 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:30.651385 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:30.652149 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:30.652464 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:30.652468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:30.652710 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:30.652914 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.652918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.653656 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.653658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.654498 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.654501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.654911 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.654912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.655055 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.655680 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:30.655691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:30.658078 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:30.660051 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.660057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.660059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.660366 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.660594 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.660595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.660595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.660719 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.661023 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:30.661029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:30.661198 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 20:24:30.661306 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 20:24:30.661470 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":124,"threads_started":1}
I20260928 20:24:30.662247 26643 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40279
I20260928 20:24:30.662328 26643 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:30.662505 26643 heartbeater.cc:511] Master 127.24.186.254:40279 requested a full tablet report, sending...
I20260928 20:24:30.662977 26439 ts_manager.cc:194] Registered new tserver with Master: fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133)
I20260928 20:24:30.663795 26439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:44873
I20260928 20:24:30.668146 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26512
I20260928 20:24:30.668282 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:30.668483 25323 external_mini_cluster.cc:1428] Setting key 44a108150f7aa61898f2b89888c757f8
I20260928 20:24:30.669687 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.663344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:51833 (local address 127.24.186.254:40279)
0928 20:24:30.663383 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:30.663385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:30.663478 (+ 93us) server_negotiation.cc:407] Connection header received
0928 20:24:30.663532 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:30.663533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:30.663542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:30.663595 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:30.663944 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.664521 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.665255 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.665455 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.665521 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.666593 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:30.666594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:30.666595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:30.666601 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:30.667940 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.668721 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.668722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.668723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.668792 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.669122 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.669123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.669123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.669261 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:30.669333 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:30.669540 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 20:24:30.669624 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:30.669770 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.663284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.193:51833)
0928 20:24:30.663405 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:30.663416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:30.663477 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:30.663599 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:30.663600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:30.663759 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:30.663869 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.663871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.664628 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.664629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.665157 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.665159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.665572 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.665573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.665639 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.666177 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:30.666183 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:30.666506 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:30.668390 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.668391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.668391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.668626 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.668978 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.668979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.668979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.669044 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.669355 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:30.669356 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:30.669411 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 20:24:30.669500 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:24:30.669627 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:30.699502 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40279
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:30.788898 26652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:30.789125 26652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:30.789152 26652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:30.790330 26652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:30.790369 26652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:30.790388 26652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:30.790406 26652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:30.791898 26652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:30.792110 26652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:30.792299 26652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:30.797442 26657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.797575 26652 server_base.cc:1034] running on GCE node
W20260928 20:24:30.797439 26660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:30.797452 26658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.797922 26652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:30.798229 26652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:30.799430 26652 hybrid_clock.cc:648] HybridClock initialized: now 1790627070799410 us; error 37 us; skew 500 ppm
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:30.803136 26652 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:30.803683 26652 webserver.cc:460] Webserver started at http://127.24.186.194:45071/ using document root <none> and password file <none>
I20260928 20:24:30.803952 26652 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:30.804001 26652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:30.804153 26652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:30.805310 26652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "5489e53c26d74b8c80e58235551d72a9"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "75bb984df36bf3880032a64c95bfdb4b"
server_key_iv: "ebc35acb6853abd8dbd054a2311fa977"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.805747 26652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "5489e53c26d74b8c80e58235551d72a9"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "75bb984df36bf3880032a64c95bfdb4b"
server_key_iv: "ebc35acb6853abd8dbd054a2311fa977"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.807117 26652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:30.808048 26667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.808424 26652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:30.808506 26652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "5489e53c26d74b8c80e58235551d72a9"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "75bb984df36bf3880032a64c95bfdb4b"
server_key_iv: "ebc35acb6853abd8dbd054a2311fa977"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.808634 26652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:30.829541 26652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:30.831568 26652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:30.831684 26652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:30.832346 26652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:30.832388 26652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.832412 26652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:30.832424 26652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.839643 26652 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:38285
I20260928 20:24:30.839699 26782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:38285 every 8 connection(s)
I20260928 20:24:30.840046 26652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:30.843654 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.832815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:35685 (local address 127.24.186.254:40279)
0928 20:24:30.832980 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:30.832983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:30.833182 (+ 199us) server_negotiation.cc:407] Connection header received
0928 20:24:30.833765 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:30.833766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:30.833780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:30.833830 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:30.834830 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.835692 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.837539 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.837797 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.837859 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.840730 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:30.840731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:30.840732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:30.840740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:30.842201 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.842646 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.842647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.842648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.842710 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.842980 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.842981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.842982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.843107 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:30.843191 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:30.843478 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 20:24:30.843561 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:24:30.843984 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.832756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.194:35685)
0928 20:24:30.833053 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:30.833090 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:30.833659 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:30.833981 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:30.833987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:30.834386 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:30.834690 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.834697 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.835857 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.835858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.837421 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.837425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.837899 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.837900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.838070 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.838750 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:30.838762 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:30.840617 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:30.842314 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.842321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.842324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.842565 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.842780 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.842780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.842781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.842901 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.843175 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:30.843182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:30.843333 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:24:30.843442 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 20:24:30.843612 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260928 20:24:30.844583 26783 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40279
I20260928 20:24:30.844679 26783 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:30.845010 26783 heartbeater.cc:511] Master 127.24.186.254:40279 requested a full tablet report, sending...
I20260928 20:24:30.845382 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26652
I20260928 20:24:30.845471 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:30.845453 26440 ts_manager.cc:194] Registered new tserver with Master: 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285)
I20260928 20:24:30.845683 25323 external_mini_cluster.cc:1428] Setting key 5f91b267d941d9a22a188c66bf95f161
I20260928 20:24:30.845961 26440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:35685
I20260928 20:24:30.852404 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.846073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:43099 (local address 127.24.186.254:40279)
0928 20:24:30.846117 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:30.846120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:30.846129 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:30.846194 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:30.846195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:30.846205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:30.846261 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:30.846664 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.847399 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.848486 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.848779 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.848869 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.849853 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:30.849855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:30.849856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:30.849863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:30.851162 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.851509 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.851510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.851511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.851567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:30.851792 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:30.851793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:30.851793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:30.851975 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:30.852053 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:30.852225 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 20:24:30.852315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:24:30.852416 26789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:30.845812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.194:43099)
0928 20:24:30.846060 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:30.846072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:30.846133 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:30.846267 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:30.846268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:30.846508 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:30.846577 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.846579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.847503 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.847504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.848407 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:30.848409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:30.848869 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:30.848870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:30.848949 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:30.849446 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:30.849451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:30.849743 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:30.851253 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.851254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.851254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.851433 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.851642 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:30.851643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:30.851643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:30.851703 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:30.852054 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:30.852056 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:30.852104 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:24:30.852184 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:24:30.852302 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":193,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:30.875324 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40279
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:30.960147 26792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:30.960369 26792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:30.960397 26792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:30.961634 26792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:30.961683 26792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:30.961706 26792 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:30.961758 26792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:30.963501 26792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:30.963757 26792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:30.964042 26792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:30.969748 26800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.969904 26792 server_base.cc:1034] running on GCE node
W20260928 20:24:30.969748 26797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:30.969825 26798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:30.970240 26792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:30.970463 26792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:30.971621 26792 hybrid_clock.cc:648] HybridClock initialized: now 1790627070971604 us; error 33 us; skew 500 ppm
Sep 28 20:24:30 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:30.974416 26792 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:30.974822 26792 webserver.cc:460] Webserver started at http://127.24.186.195:37933/ using document root <none> and password file <none>
I20260928 20:24:30.975096 26792 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:30.975145 26792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:30.975301 26792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:30.976435 26792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "c385a450f6ec9c5a536eecb81976880b"
server_key_iv: "39261d1c2c6ef918fd8f2e592a320be4"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.976780 26792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "c385a450f6ec9c5a536eecb81976880b"
server_key_iv: "39261d1c2c6ef918fd8f2e592a320be4"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.978148 26792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:24:30.978976 26807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:30.979197 26792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:30.979297 26792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4"
format_stamp: "Formatted at 2026-09-28 20:24:30 on dist-test-slave-k477"
server_key: "c385a450f6ec9c5a536eecb81976880b"
server_key_iv: "39261d1c2c6ef918fd8f2e592a320be4"
server_key_version: "encryptionkey@0"
I20260928 20:24:30.979357 26792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:30.999089 26792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:31.001385 26792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:31.001510 26792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:31.002018 26792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:31.002061 26792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:31.002086 26792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:31.002115 26792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:31.009224 26792 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:36451
I20260928 20:24:31.009294 26922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:36451 every 8 connection(s)
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:31.009657 26792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:31.010782 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 26792
I20260928 20:24:31.010879 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:31.011058 25323 external_mini_cluster.cc:1428] Setting key e9af8e7adcc6b6707944c692335ca221
I20260928 20:24:31.013599 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.002535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:33913 (local address 127.24.186.254:40279)
0928 20:24:31.002612 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.002615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.002773 (+ 158us) server_negotiation.cc:407] Connection header received
0928 20:24:31.003271 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.003272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.003285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.003363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.004218 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.005065 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.006636 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.006964 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.007048 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.010128 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.010131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.010131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.010140 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.012261 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.012755 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.012756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.012756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.012816 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.013075 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.013076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.013076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.013183 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.013233 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.013470 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.013535 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:24:31.013801 26816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.002490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.195:33913)
0928 20:24:31.002643 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.002684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.003158 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.003441 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.003446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.003834 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:31.004115 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.004122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.005166 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.005167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.006531 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.006535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.007066 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.007067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.007188 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.007781 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:31.007794 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:31.010017 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:31.012346 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.012352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.012353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.012691 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.012902 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.012903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.012903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.013014 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.013225 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:31.013235 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:31.013354 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.013445 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.013577 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260928 20:24:31.014271 26923 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40279
I20260928 20:24:31.014345 26923 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:31.014555 26923 heartbeater.cc:511] Master 127.24.186.254:40279 requested a full tablet report, sending...
I20260928 20:24:31.014922 26440 ts_manager.cc:194] Registered new tserver with Master: 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451)
I20260928 20:24:31.015482 26440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:33913
I20260928 20:24:31.021745 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.015358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:40469 (local address 127.24.186.254:40279)
0928 20:24:31.015393 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.015396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.015743 (+ 347us) server_negotiation.cc:407] Connection header received
0928 20:24:31.015785 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.015786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.015797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.015877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.016264 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.016854 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.017595 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.017818 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.017888 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.019233 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.019235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.019235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.019244 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.020553 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.020943 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.020945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.020945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.021000 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.021203 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.021204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.021204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.021318 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.021397 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.021574 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.021651 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:24:31.021790 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.015292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.24.186.195:40469)
0928 20:24:31.015680 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.015690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.015744 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.015949 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.015951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.016129 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:31.016187 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.016189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.016942 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.016943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.017513 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.017515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.017898 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.017899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.017964 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.018535 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:31.018543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:31.018990 (+ 447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:31.020643 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.020645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.020645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.020855 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.021073 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.021074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.021075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.021140 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.021388 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:31.021389 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:31.021456 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.021535 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.021655 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":61,"threads_started":1}
I20260928 20:24:31.022774 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[26395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:31.046887 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.039419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.24.186.254:40279)
0928 20:24:31.039484 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.039487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.039515 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:24:31.039550 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.039551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.039560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.039618 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.040281 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.040783 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.041528 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.041724 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.041814 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.044204 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.044206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.044211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.044218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.045636 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.046062 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.046063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.046064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.046123 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.046331 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.046332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.046332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.046453 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.046558 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.046699 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.046781 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:24:31.048594 26440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:31.050117 26440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:31.056456 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:38285 (local address 127.0.0.1:51708)
0928 20:24:31.052683 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.052702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.052744 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.052990 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.052993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.052995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.053047 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.053049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.054266 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.054267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.055098 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.055101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.056124 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.056125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.056216 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.056230 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.056305 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.056368 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260928 20:24:31.056510 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.24.186.194:38285)
0928 20:24:31.052760 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.052766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.052778 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:31.052823 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.052825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.052842 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.052917 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.053149 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.054154 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.055254 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.056026 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.056124 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.056192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.056364 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.056403 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260928 20:24:31.057291 26717 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f (DEFAULT_TABLE table=test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.057679 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36451 (local address 127.0.0.1:58422)
0928 20:24:31.052530 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.052573 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.052641 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.053404 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.053405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.053407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.053460 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.053462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.055260 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.055261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.056508 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.056510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.057424 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.057425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.057497 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.057502 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.057555 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.057603 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":76,"spinlock_wait_cycles":768,"thread_start_us":90,"threads_started":1}
I20260928 20:24:31.057685 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c41510d2f470e893c27ff8e1de88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.057924 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.24.186.195:36451)
0928 20:24:31.053204 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.053208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.053217 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:31.053255 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.053257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.053272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.053342 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.053539 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.055179 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.056745 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.057319 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.057422 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.057474 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.057778 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.057827 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":92,"threads_started":1}
I20260928 20:24:31.058123 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38133 (local address 127.0.0.1:36548)
0928 20:24:31.052787 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.052798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.052854 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.053141 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.053142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.053144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.053267 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.053269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.055227 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.055228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.056495 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.056498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.057863 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.057864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.057936 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.057940 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.057989 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.058034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260928 20:24:31.058384 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.052794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.24.186.193:38133)
0928 20:24:31.052960 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.052964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.052973 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:31.053009 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.053012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.053029 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.053090 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.053358 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.055105 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.056976 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.057700 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.057774 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.057825 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.058247 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.058291 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260928 20:24:31.059029 26577 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f (DEFAULT_TABLE table=test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.059382 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c41510d2f470e893c27ff8e1de88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.060516 26857 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f (DEFAULT_TABLE table=test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.060851 26857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7c41510d2f470e893c27ff8e1de88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.061623 26946 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Bootstrap starting.
I20260928 20:24:31.062387 26946 tablet_bootstrap.cc:654] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.062816 26946 log.cc:824] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:31.062883 26947 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Bootstrap starting.
I20260928 20:24:31.063599 26947 tablet_bootstrap.cc:654] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.063774 26946 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: No bootstrap required, opened a new log
I20260928 20:24:31.063978 26947 log.cc:824] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:31.064142 26946 ts_tablet_manager.cc:1397] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:24:31.064711 26947 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: No bootstrap required, opened a new log
I20260928 20:24:31.064785 26947 ts_tablet_manager.cc:1397] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:24:31.064780 26950 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Bootstrap starting.
I20260928 20:24:31.065754 26950 tablet_bootstrap.cc:654] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.066066 26950 log.cc:824] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:31.066082 26947 raft_consensus.cc:348] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.066198 26947 raft_consensus.cc:374] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.066229 26947 raft_consensus.cc:729] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd595e996fc44805ab40dffe424e90bc, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.066357 26947 consensus_queue.cc:260] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.066625 26947 ts_tablet_manager.cc:1428] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:24:31.066757 26643 heartbeater.cc:503] Master 127.24.186.254:40279 was elected leader, sending a full tablet report...
I20260928 20:24:31.066699 26946 raft_consensus.cc:348] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.066861 26946 raft_consensus.cc:374] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.066875 26950 tablet_bootstrap.cc:492] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: No bootstrap required, opened a new log
I20260928 20:24:31.066900 26946 raft_consensus.cc:729] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5489e53c26d74b8c80e58235551d72a9, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.066927 26950 ts_tablet_manager.cc:1397] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:31.067014 26946 consensus_queue.cc:260] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.067274 26946 ts_tablet_manager.cc:1428] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:31.067327 26783 heartbeater.cc:503] Master 127.24.186.254:40279 was elected leader, sending a full tablet report...
I20260928 20:24:31.069132 26950 raft_consensus.cc:348] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.069250 26950 raft_consensus.cc:374] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.069271 26950 raft_consensus.cc:729] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7fb52fd4564ce1ac921da3ebdf3dd4, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.069377 26950 consensus_queue.cc:260] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.069576 26950 ts_tablet_manager.cc:1428] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:24:31.069650 26923 heartbeater.cc:503] Master 127.24.186.254:40279 was elected leader, sending a full tablet report...
W20260928 20:24:31.090781 26784 tablet.cc:2369] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:31.127617 26953 raft_consensus.cc:482] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:31.127753 26953 raft_consensus.cc:504] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.128165 26953 leader_election.cc:288] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451), fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133)
I20260928 20:24:31.131310 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.128500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:43731 (local address 127.24.186.195:36451)
0928 20:24:31.128559 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.128561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.128570 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:31.128602 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.128603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.128611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.128678 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.128833 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.129394 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.130489 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.130951 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.131035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.131045 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.131205 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.131243 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:24:31.131323 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.128391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36451 (local address 127.24.186.194:43731)
0928 20:24:31.128445 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.128458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.128501 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.128683 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.128684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.128686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.128752 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.128754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.129518 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.129519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.130407 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.130410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.131032 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.131033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.131114 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.131118 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.131196 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.131238 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":18}
I20260928 20:24:31.131649 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c41510d2f470e893c27ff8e1de88f" candidate_uuid: "5489e53c26d74b8c80e58235551d72a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" is_pre_election: true
I20260928 20:24:31.131856 26877 raft_consensus.cc:2454] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5489e53c26d74b8c80e58235551d72a9 in term 0.
I20260928 20:24:31.132094 26668 leader_election.cc:302] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7fb52fd4564ce1ac921da3ebdf3dd4, 5489e53c26d74b8c80e58235551d72a9; no voters:
I20260928 20:24:31.132169 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.128413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:60595 (local address 127.24.186.193:38133)
0928 20:24:31.128455 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.128458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.128711 (+ 253us) server_negotiation.cc:407] Connection header received
0928 20:24:31.128780 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.128781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.128790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.128855 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.129009 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.129471 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.131209 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.131755 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.131869 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.131879 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.132059 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.132103 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:24:31.132242 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.128393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38133 (local address 127.24.186.194:60595)
0928 20:24:31.128627 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.128638 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.128724 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.128858 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.128859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.128860 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.128931 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.128933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.129557 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.129558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.131088 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.131091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.131907 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.131909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.131994 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.131999 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.132052 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.132093 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":53,"thread_start_us":113,"threads_started":1}
I20260928 20:24:31.132256 26953 raft_consensus.cc:2781] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:31.132329 26953 raft_consensus.cc:482] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:31.132359 26953 raft_consensus.cc:3037] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.132622 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c41510d2f470e893c27ff8e1de88f" candidate_uuid: "5489e53c26d74b8c80e58235551d72a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd595e996fc44805ab40dffe424e90bc" is_pre_election: true
I20260928 20:24:31.132766 26597 raft_consensus.cc:2454] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5489e53c26d74b8c80e58235551d72a9 in term 0.
I20260928 20:24:31.133450 26953 raft_consensus.cc:504] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.133600 26953 leader_election.cc:288] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [CANDIDATE]: Term 1 election: Requested vote from peers 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451), fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133)
I20260928 20:24:31.133747 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c41510d2f470e893c27ff8e1de88f" candidate_uuid: "5489e53c26d74b8c80e58235551d72a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd595e996fc44805ab40dffe424e90bc"
I20260928 20:24:31.133762 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7c41510d2f470e893c27ff8e1de88f" candidate_uuid: "5489e53c26d74b8c80e58235551d72a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4"
I20260928 20:24:31.133823 26597 raft_consensus.cc:3037] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.133854 26877 raft_consensus.cc:3037] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.134519 26877 raft_consensus.cc:2454] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5489e53c26d74b8c80e58235551d72a9 in term 1.
I20260928 20:24:31.134526 26597 raft_consensus.cc:2454] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5489e53c26d74b8c80e58235551d72a9 in term 1.
I20260928 20:24:31.134687 26668 leader_election.cc:302] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7fb52fd4564ce1ac921da3ebdf3dd4, 5489e53c26d74b8c80e58235551d72a9; no voters:
I20260928 20:24:31.134796 26953 raft_consensus.cc:2781] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:31.134955 26953 raft_consensus.cc:686] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 1 LEADER]: Becoming Leader. State: Replica: 5489e53c26d74b8c80e58235551d72a9, State: Running, Role: LEADER
I20260928 20:24:31.135057 26953 consensus_queue.cc:237] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.135931 26439 catalog_manager.cc:5495] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "5489e53c26d74b8c80e58235551d72a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:31.155081 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.152541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.24.186.194:38285)
0928 20:24:31.152605 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.152607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.152618 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:31.152716 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.152717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.152743 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.152817 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:31.152960 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.153450 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.154246 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.154443 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.154519 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.154727 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.154838 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.154928 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.154973 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.154995 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:31.157145 26597 raft_consensus.cc:1261] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Refusing update from remote peer 5489e53c26d74b8c80e58235551d72a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.157150 26877 raft_consensus.cc:1261] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Refusing update from remote peer 5489e53c26d74b8c80e58235551d72a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.157598 26956 consensus_queue.cc:1035] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.157793 26953 consensus_queue.cc:1035] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.160218 26959 mvcc.cc:205] Tried to move back new op lower bound from 7334408483456417792 to 7334408483369779200. Current Snapshot: MvccSnapshot[applied={T|T < 7334408483456417792}]
I20260928 20:24:31.160655 26961 mvcc.cc:205] Tried to move back new op lower bound from 7334408483456417792 to 7334408483369779200. Current Snapshot: MvccSnapshot[applied={T|T < 7334408483456417792}]
I20260928 20:24:31.161157 26960 mvcc.cc:205] Tried to move back new op lower bound from 7334408483456417792 to 7334408483369779200. Current Snapshot: MvccSnapshot[applied={T|T < 7334408483456417792}]
I20260928 20:24:31.164615 26440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:31.164728 26440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:31.165632 26440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Sending DeleteTablet for 3 replicas of tablet 1d7c41510d2f470e893c27ff8e1de88f
I20260928 20:24:31.165946 26857 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:58422
I20260928 20:24:31.165980 26717 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:51708
I20260928 20:24:31.166183 26577 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7c41510d2f470e893c27ff8e1de88f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:36548
I20260928 20:24:31.166306 26968 tablet_replica.cc:330] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: stopping tablet replica
I20260928 20:24:31.166416 26968 raft_consensus.cc:2229] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:31.166527 26969 tablet_replica.cc:330] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: stopping tablet replica
I20260928 20:24:31.166620 26969 raft_consensus.cc:2229] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:31.166625 26440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 20:24:31.166751 26969 raft_consensus.cc:2258] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 20:24:31.166780 26440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:31.167222 26969 ts_tablet_manager.cc:1905] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.167999 26577 tablet_service.cc:1467] Processing CreateTablet for tablet 61fedeef49f246d388c31eeae513fc7c (DEFAULT_TABLE table=test-table [id=3ff1acd1262e49a18ee13375131bb54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.168068 26717 tablet_service.cc:1467] Processing CreateTablet for tablet 61fedeef49f246d388c31eeae513fc7c (DEFAULT_TABLE table=test-table [id=3ff1acd1262e49a18ee13375131bb54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.168133 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61fedeef49f246d388c31eeae513fc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.168267 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61fedeef49f246d388c31eeae513fc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.169067 26969 ts_tablet_manager.cc:1918] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:31.169142 26969 log.cc:1196] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/1d7c41510d2f470e893c27ff8e1de88f
I20260928 20:24:31.169443 26947 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Bootstrap starting.
I20260928 20:24:31.169504 26857 tablet_service.cc:1467] Processing CreateTablet for tablet 61fedeef49f246d388c31eeae513fc7c (DEFAULT_TABLE table=test-table [id=3ff1acd1262e49a18ee13375131bb54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:31.169615 26857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61fedeef49f246d388c31eeae513fc7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.169840 26947 tablet_bootstrap.cc:654] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.169396 26969 ts_tablet_manager.cc:1939] T 1d7c41510d2f470e893c27ff8e1de88f P 5489e53c26d74b8c80e58235551d72a9: Deleting consensus metadata
I20260928 20:24:31.170675 26950 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Bootstrap starting.
I20260928 20:24:31.170710 26946 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Bootstrap starting.
I20260928 20:24:31.170743 26947 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: No bootstrap required, opened a new log
I20260928 20:24:31.170786 26947 ts_tablet_manager.cc:1397] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:31.170894 26947 raft_consensus.cc:348] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.170960 26947 raft_consensus.cc:374] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.170981 26947 raft_consensus.cc:729] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd595e996fc44805ab40dffe424e90bc, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.171029 26947 consensus_queue.cc:260] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.171095 26946 tablet_bootstrap.cc:654] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.171095 26950 tablet_bootstrap.cc:654] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.171108 26947 ts_tablet_manager.cc:1428] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:31.172048 26968 raft_consensus.cc:2258] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:31.172133 26426 catalog_manager.cc:4841] TS 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285): tablet 1d7c41510d2f470e893c27ff8e1de88f (table test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]) successfully deleted
I20260928 20:24:31.172219 26970 tablet_replica.cc:330] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: stopping tablet replica
I20260928 20:24:31.172346 26970 raft_consensus.cc:2229] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:31.172369 26946 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: No bootstrap required, opened a new log
I20260928 20:24:31.172416 26946 ts_tablet_manager.cc:1397] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:31.172489 26970 raft_consensus.cc:2258] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:31.172559 26946 raft_consensus.cc:348] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.172640 26946 raft_consensus.cc:374] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.172672 26946 raft_consensus.cc:729] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5489e53c26d74b8c80e58235551d72a9, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.172712 26946 consensus_queue.cc:260] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.172797 26946 ts_tablet_manager.cc:1428] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:31.173125 26968 ts_tablet_manager.cc:1905] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.173362 26970 ts_tablet_manager.cc:1905] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.175022 26968 ts_tablet_manager.cc:1918] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:31.175135 26968 log.cc:1196] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/1d7c41510d2f470e893c27ff8e1de88f
I20260928 20:24:31.175420 26968 ts_tablet_manager.cc:1939] T 1d7c41510d2f470e893c27ff8e1de88f P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting consensus metadata
I20260928 20:24:31.175572 26970 ts_tablet_manager.cc:1918] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:31.175648 26970 log.cc:1196] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/1d7c41510d2f470e893c27ff8e1de88f
I20260928 20:24:31.176179 26970 ts_tablet_manager.cc:1939] T 1d7c41510d2f470e893c27ff8e1de88f P fd595e996fc44805ab40dffe424e90bc: Deleting consensus metadata
I20260928 20:24:31.176367 26424 catalog_manager.cc:4841] TS 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451): tablet 1d7c41510d2f470e893c27ff8e1de88f (table test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]) successfully deleted
I20260928 20:24:31.176621 26426 catalog_manager.cc:4841] TS fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133): tablet 1d7c41510d2f470e893c27ff8e1de88f (table test-table [id=fe4a1a3e0e1a447eb40fb93a4357e559]) successfully deleted
I20260928 20:24:31.176892 26950 tablet_bootstrap.cc:492] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: No bootstrap required, opened a new log
I20260928 20:24:31.176935 26950 ts_tablet_manager.cc:1397] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 20:24:31.177104 26950 raft_consensus.cc:348] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.177172 26950 raft_consensus.cc:374] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.177186 26950 raft_consensus.cc:729] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7fb52fd4564ce1ac921da3ebdf3dd4, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.177241 26950 consensus_queue.cc:260] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.177333 26950 ts_tablet_manager.cc:1428] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260928 20:24:31.260336 26924 tablet.cc:2369] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:31.411791 26644 tablet.cc:2369] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:31.525691 26954 raft_consensus.cc:482] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:31.525839 26954 raft_consensus.cc:504] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.526219 26954 leader_election.cc:288] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133), 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285)
I20260928 20:24:31.530273 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.526764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:33923 (local address 127.24.186.194:38285)
0928 20:24:31.526822 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.526824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.526833 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:31.526876 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.526877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.526888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.526937 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.527076 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.527835 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529339 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.529895 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529970 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.529978 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.530157 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.530203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:24:31.530273 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.526538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:49535 (local address 127.24.186.193:38133)
0928 20:24:31.526588 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.526590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.526703 (+ 113us) server_negotiation.cc:407] Connection header received
0928 20:24:31.526778 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.526779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.526788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.526841 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.527062 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.527841 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529338 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.529891 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529970 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.529978 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.530157 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.530199 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:31.530334 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.526469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38133 (local address 127.24.186.195:49535)
0928 20:24:31.526617 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.526627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.526721 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.526860 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.526861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.526863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.526966 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.526968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.527919 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.527920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.529219 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.529222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.529987 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.530071 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.530073 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.530134 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.530188 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":96,"thread_start_us":63,"threads_started":1}
I20260928 20:24:31.530342 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.526630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:38285 (local address 127.24.186.195:33923)
0928 20:24:31.526767 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.526777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.526826 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.526960 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.526961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.526962 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.527003 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.527004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.527919 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.527920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.529219 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.529222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.529989 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.529989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.530068 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.530072 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.530134 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.530188 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":35,"thread_start_us":36,"threads_started":1}
I20260928 20:24:31.530609 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fedeef49f246d388c31eeae513fc7c" candidate_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd595e996fc44805ab40dffe424e90bc" is_pre_election: true
I20260928 20:24:31.530723 26597 raft_consensus.cc:2454] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b7fb52fd4564ce1ac921da3ebdf3dd4 in term 0.
I20260928 20:24:31.530660 26737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fedeef49f246d388c31eeae513fc7c" candidate_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489e53c26d74b8c80e58235551d72a9" is_pre_election: true
I20260928 20:24:31.530820 26737 raft_consensus.cc:2454] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b7fb52fd4564ce1ac921da3ebdf3dd4 in term 0.
I20260928 20:24:31.530964 26810 leader_election.cc:302] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7fb52fd4564ce1ac921da3ebdf3dd4, fd595e996fc44805ab40dffe424e90bc; no voters:
I20260928 20:24:31.531162 26954 raft_consensus.cc:2781] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:31.531219 26954 raft_consensus.cc:482] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:31.531248 26954 raft_consensus.cc:3037] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.532038 26954 raft_consensus.cc:504] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.532184 26954 leader_election.cc:288] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [CANDIDATE]: Term 1 election: Requested vote from peers fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133), 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285)
I20260928 20:24:31.532429 26597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fedeef49f246d388c31eeae513fc7c" candidate_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd595e996fc44805ab40dffe424e90bc"
I20260928 20:24:31.532449 26737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61fedeef49f246d388c31eeae513fc7c" candidate_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489e53c26d74b8c80e58235551d72a9"
I20260928 20:24:31.532526 26597 raft_consensus.cc:3037] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.532524 26737 raft_consensus.cc:3037] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.533324 26597 raft_consensus.cc:2454] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7fb52fd4564ce1ac921da3ebdf3dd4 in term 1.
I20260928 20:24:31.533373 26737 raft_consensus.cc:2454] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b7fb52fd4564ce1ac921da3ebdf3dd4 in term 1.
I20260928 20:24:31.533514 26810 leader_election.cc:302] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7fb52fd4564ce1ac921da3ebdf3dd4, fd595e996fc44805ab40dffe424e90bc; no voters:
I20260928 20:24:31.533622 26954 raft_consensus.cc:2781] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:31.533740 26954 raft_consensus.cc:686] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 3b7fb52fd4564ce1ac921da3ebdf3dd4, State: Running, Role: LEADER
I20260928 20:24:31.533954 26954 consensus_queue.cc:237] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } }
I20260928 20:24:31.534725 26439 catalog_manager.cc:5495] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195). New cstate: current_term: 1 leader_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } health_report { overall_health: HEALTHY } } }
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627070, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:31.652107 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.644676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54576 (local address 127.24.186.254:40279)
0928 20:24:31.644852 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.644854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.644882 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:24:31.645039 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.645040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.645052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.645129 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.645801 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.646324 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.647062 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.647271 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.647340 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.649518 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.649520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.649520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.649527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.650852 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.651291 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.651292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.651293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.651356 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.651539 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.651540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.651540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.651654 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.651707 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.651927 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.652029 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260928 20:24:31.652185 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.644632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40279 (local address 127.0.0.1:54576)
0928 20:24:31.644816 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.644834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.644911 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.645190 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.645191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.645581 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:31.645666 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.645669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.646417 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.646418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.646957 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.646959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.647369 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.647370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.647418 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.648049 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:31.648053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:31.649410 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:31.650951 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.650958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.650960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.651194 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.651426 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:31.651427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:31.651427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:31.651483 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:31.651719 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:31.651725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:31.651768 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.651903 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.652097 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":88,"thread_start_us":94,"threads_started":1}
I20260928 20:24:31.654424 26439 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54576:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:24:31.654611 26439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:31.656220 26717 tablet_service.cc:1467] Processing CreateTablet for tablet ea215def09ec4065ab16e74c7ead0d06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=132c5eff9993472b9564053c7149a4c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:31.656257 26577 tablet_service.cc:1467] Processing CreateTablet for tablet ea215def09ec4065ab16e74c7ead0d06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=132c5eff9993472b9564053c7149a4c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:31.656262 26857 tablet_service.cc:1467] Processing CreateTablet for tablet ea215def09ec4065ab16e74c7ead0d06 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=132c5eff9993472b9564053c7149a4c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:31.656407 26577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea215def09ec4065ab16e74c7ead0d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.656407 26717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea215def09ec4065ab16e74c7ead0d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.656445 26857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea215def09ec4065ab16e74c7ead0d06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:31.657559 26950 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Bootstrap starting.
I20260928 20:24:31.657893 26947 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: Bootstrap starting.
I20260928 20:24:31.657903 26946 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: Bootstrap starting.
I20260928 20:24:31.658018 26950 tablet_bootstrap.cc:654] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.658278 26946 tablet_bootstrap.cc:654] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.658352 26947 tablet_bootstrap.cc:654] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:31.658783 26950 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: No bootstrap required, opened a new log
I20260928 20:24:31.658844 26950 ts_tablet_manager.cc:1397] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:31.659032 26947 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: No bootstrap required, opened a new log
I20260928 20:24:31.659085 26947 ts_tablet_manager.cc:1397] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:31.659032 26950 raft_consensus.cc:348] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.659121 26950 raft_consensus.cc:374] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.659153 26950 raft_consensus.cc:729] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7fb52fd4564ce1ac921da3ebdf3dd4, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.659212 26950 consensus_queue.cc:260] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.659236 26947 raft_consensus.cc:348] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.659307 26950 ts_tablet_manager.cc:1428] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:31.659333 26947 raft_consensus.cc:374] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.659366 26947 raft_consensus.cc:729] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd595e996fc44805ab40dffe424e90bc, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.659415 26947 consensus_queue.cc:260] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.659456 26954 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.659518 26947 ts_tablet_manager.cc:1428] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:31.659534 26954 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.659650 26952 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.659737 26952 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.660038 26946 tablet_bootstrap.cc:492] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: No bootstrap required, opened a new log
I20260928 20:24:31.660208 26946 ts_tablet_manager.cc:1397] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:31.660351 26946 raft_consensus.cc:348] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.660450 26946 raft_consensus.cc:374] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:31.660476 26946 raft_consensus.cc:729] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5489e53c26d74b8c80e58235551d72a9, State: Initialized, Role: FOLLOWER
I20260928 20:24:31.660527 26946 consensus_queue.cc:260] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.660614 26946 ts_tablet_manager.cc:1428] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:31.660807 26953 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.660892 26953 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.676800 26952 raft_consensus.cc:482] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:31.676901 26952 raft_consensus.cc:504] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.677286 26952 leader_election.cc:288] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285), 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451)
I20260928 20:24:31.681231 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.677519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:59595 (local address 127.24.186.194:38285)
0928 20:24:31.677578 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.677581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.677662 (+ 81us) server_negotiation.cc:407] Connection header received
0928 20:24:31.677697 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.677698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.677709 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.677761 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.677997 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.678750 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680264 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.680826 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680907 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.680916 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.681110 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.681149 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:24:31.681380 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.677672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:43209 (local address 127.24.186.195:36451)
0928 20:24:31.677767 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.677769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.677776 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:31.677797 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.677798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.677805 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.677862 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:31.677996 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.678744 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680265 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.680835 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680908 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.680916 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.681116 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.681298 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":48,"threads_started":1}
I20260928 20:24:31.681239 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.677457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:38285 (local address 127.24.186.193:59595)
0928 20:24:31.677563 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.677576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.677646 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.677817 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.677818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.677820 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.677903 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.677904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.678834 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.678834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.680175 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.680178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.680937 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.681020 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.681023 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.681093 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.681142 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":68,"thread_start_us":51,"threads_started":1}
I20260928 20:24:31.681277 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.677516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36451 (local address 127.24.186.193:43209)
0928 20:24:31.677606 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.677617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.677687 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.677884 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.677885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.677886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:31.677922 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.677924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.678831 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.678832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.680175 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.680178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.680938 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.680939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.681017 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.681022 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.681166 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.681200 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":83,"thread_start_us":52,"threads_started":1}
I20260928 20:24:31.681720 26737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea215def09ec4065ab16e74c7ead0d06" candidate_uuid: "fd595e996fc44805ab40dffe424e90bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489e53c26d74b8c80e58235551d72a9" is_pre_election: true
I20260928 20:24:31.681829 26737 raft_consensus.cc:2454] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd595e996fc44805ab40dffe424e90bc in term 0.
I20260928 20:24:31.681865 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea215def09ec4065ab16e74c7ead0d06" candidate_uuid: "fd595e996fc44805ab40dffe424e90bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" is_pre_election: true
I20260928 20:24:31.681962 26877 raft_consensus.cc:2454] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd595e996fc44805ab40dffe424e90bc in term 0.
I20260928 20:24:31.682030 26530 leader_election.cc:302] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5489e53c26d74b8c80e58235551d72a9, fd595e996fc44805ab40dffe424e90bc; no voters:
I20260928 20:24:31.682265 26952 raft_consensus.cc:2781] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:31.682300 26952 raft_consensus.cc:482] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:31.682314 26952 raft_consensus.cc:3037] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.682984 26952 raft_consensus.cc:504] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.683131 26952 leader_election.cc:288] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [CANDIDATE]: Term 1 election: Requested vote from peers 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285), 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451)
I20260928 20:24:31.683245 26737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea215def09ec4065ab16e74c7ead0d06" candidate_uuid: "fd595e996fc44805ab40dffe424e90bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5489e53c26d74b8c80e58235551d72a9"
I20260928 20:24:31.683316 26737 raft_consensus.cc:3037] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.683281 26877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea215def09ec4065ab16e74c7ead0d06" candidate_uuid: "fd595e996fc44805ab40dffe424e90bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4"
I20260928 20:24:31.683362 26877 raft_consensus.cc:3037] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:31.683949 26737 raft_consensus.cc:2454] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd595e996fc44805ab40dffe424e90bc in term 1.
I20260928 20:24:31.684089 26877 raft_consensus.cc:2454] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd595e996fc44805ab40dffe424e90bc in term 1.
I20260928 20:24:31.684101 26530 leader_election.cc:302] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5489e53c26d74b8c80e58235551d72a9, fd595e996fc44805ab40dffe424e90bc; no voters:
I20260928 20:24:31.684262 26952 raft_consensus.cc:2781] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:31.684403 26952 raft_consensus.cc:686] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [term 1 LEADER]: Becoming Leader. State: Replica: fd595e996fc44805ab40dffe424e90bc, State: Running, Role: LEADER
I20260928 20:24:31.684490 26952 consensus_queue.cc:237] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } }
I20260928 20:24:31.684834 26991 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: TxnStatusTablet state changed. Reason: New leader fd595e996fc44805ab40dffe424e90bc. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.684933 26991 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:31.685128 26993 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:24:31.685237 26993 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:24:31.685191 26440 catalog_manager.cc:5495] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc reported cstate change: term changed from 0 to 1, leader changed from <none> to fd595e996fc44805ab40dffe424e90bc (127.24.186.193). New cstate: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } health_report { overall_health: HEALTHY } } }
I20260928 20:24:31.692504 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.690021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.24.186.193:38133)
0928 20:24:31.690080 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.690082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.690090 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:31.690162 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.690162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.690183 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.690243 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:31.690378 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.690854 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.691609 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.691862 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.691931 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.692121 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.692236 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.692303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.692413 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.692432 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:31.692530 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.689908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38133 (local address 127.0.0.1:36562)
0928 20:24:31.690003 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.690016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.690075 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.690244 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.690245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.690246 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:31.690299 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.690302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.690942 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.690943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.691520 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.691522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.691933 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.691934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.691992 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.692016 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.692329 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.692337 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.692408 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.692457 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 20:24:31.693849 26877 raft_consensus.cc:1261] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Refusing update from remote peer fd595e996fc44805ab40dffe424e90bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.693956 26737 raft_consensus.cc:1261] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Refusing update from remote peer fd595e996fc44805ab40dffe424e90bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.694101 26952 consensus_queue.cc:1035] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.694205 26991 consensus_queue.cc:1035] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.695267 26953 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: TxnStatusTablet state changed. Reason: New leader fd595e996fc44805ab40dffe424e90bc. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.695389 26953 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.695593 26991 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.695675 26991 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:31.695690 26954 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: TxnStatusTablet state changed. Reason: New leader fd595e996fc44805ab40dffe424e90bc. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.695782 26954 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.696915 26953 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.697074 26953 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 5489e53c26d74b8c80e58235551d72a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.698378 26952 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.698477 26952 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P fd595e996fc44805ab40dffe424e90bc: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:31.698961 26954 tablet_replica.cc:439] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd595e996fc44805ab40dffe424e90bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 } } peers { permanent_uuid: "3b7fb52fd4564ce1ac921da3ebdf3dd4" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36451 } } peers { permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 } } }
I20260928 20:24:31.699044 26954 tablet_replica.cc:442] T ea215def09ec4065ab16e74c7ead0d06 P 3b7fb52fd4564ce1ac921da3ebdf3dd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:31.702189 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.697969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.24.186.195:36451)
0928 20:24:31.698010 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.698013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.698115 (+ 102us) server_negotiation.cc:407] Connection header received
0928 20:24:31.698265 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.698266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.698286 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.698347 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:31.698550 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.700054 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.701046 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.701333 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.701468 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.701684 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.701842 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.701946 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.701990 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.702024 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:31.709034 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.705791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:47201 (local address 127.24.186.193:38133)
0928 20:24:31.705842 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.705844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.705915 (+ 71us) server_negotiation.cc:407] Connection header received
0928 20:24:31.705988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.705989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.706001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.706067 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:31.706247 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.706918 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.708290 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.708492 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.708572 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.708757 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.708795 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.708858 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.708921 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.708948 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:24:31.709059 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.705682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38133 (local address 127.24.186.195:47201)
0928 20:24:31.705853 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.705865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.705934 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.706083 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.706083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.706085 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:31.706168 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.706170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.707033 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.707034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.708183 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.708186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.708570 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.708571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.708644 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.708658 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.708861 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.708863 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.708916 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.708962 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 20:24:31.711825 26737 raft_consensus.cc:1261] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Refusing update from remote peer 3b7fb52fd4564ce1ac921da3ebdf3dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.711846 26597 raft_consensus.cc:1261] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Refusing update from remote peer 3b7fb52fd4564ce1ac921da3ebdf3dd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:31.712078 26976 consensus_queue.cc:1035] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5489e53c26d74b8c80e58235551d72a9" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.712239 26954 consensus_queue.cc:1035] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd595e996fc44805ab40dffe424e90bc" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:31.720803 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.718051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48517 (local address 127.24.186.195:36451)
0928 20:24:31.718125 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.718127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.718156 (+ 29us) server_negotiation.cc:407] Connection header received
0928 20:24:31.718177 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.718178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.718188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.718227 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:31.718384 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.718935 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.720077 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.720285 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.720360 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.720533 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.720567 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.720632 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.720698 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.720720 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:24:31.720937 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.717835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36451 (local address 127.24.186.193:48517)
0928 20:24:31.718017 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:31.718028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:31.718100 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:31.718238 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:31.718239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:31.718241 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:31.718312 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.718313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.719026 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.719027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.719993 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:31.719995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.720357 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.720358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:31.720428 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.720452 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:31.720645 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:31.720653 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:24:31.720696 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 20:24:31.720739 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260928 20:24:31.730427 26440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:31.730558 26440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:31.731237 26440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d741867978e4d0398d853dcf3f2d53d: Sending DeleteTablet for 3 replicas of tablet 61fedeef49f246d388c31eeae513fc7c
I20260928 20:24:31.731539 26577 tablet_service.cc:1514] Processing DeleteTablet for tablet 61fedeef49f246d388c31eeae513fc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:36548
I20260928 20:24:31.731600 26857 tablet_service.cc:1514] Processing DeleteTablet for tablet 61fedeef49f246d388c31eeae513fc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:58422
I20260928 20:24:31.731868 27017 tablet_replica.cc:330] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: stopping tablet replica
I20260928 20:24:31.731894 26717 tablet_service.cc:1514] Processing DeleteTablet for tablet 61fedeef49f246d388c31eeae513fc7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:31 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:51708
I20260928 20:24:31.731932 27017 raft_consensus.cc:2229] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:31.731971 27017 raft_consensus.cc:2258] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:31.732095 27019 tablet_replica.cc:330] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: stopping tablet replica
I20260928 20:24:31.732154 27019 raft_consensus.cc:2229] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:31.732195 27019 raft_consensus.cc:2258] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:31.732499 27019 ts_tablet_manager.cc:1905] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.732878 27017 ts_tablet_manager.cc:1905] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.734226 27019 ts_tablet_manager.cc:1918] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:31.734344 27019 log.cc:1196] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/61fedeef49f246d388c31eeae513fc7c
I20260928 20:24:31.734579 27019 ts_tablet_manager.cc:1939] T 61fedeef49f246d388c31eeae513fc7c P 5489e53c26d74b8c80e58235551d72a9: Deleting consensus metadata
I20260928 20:24:31.734589 27017 ts_tablet_manager.cc:1918] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:31.734656 27017 log.cc:1196] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/61fedeef49f246d388c31eeae513fc7c
I20260928 20:24:31.734908 27017 ts_tablet_manager.cc:1939] T 61fedeef49f246d388c31eeae513fc7c P fd595e996fc44805ab40dffe424e90bc: Deleting consensus metadata
I20260928 20:24:31.734999 26426 catalog_manager.cc:4841] TS 5489e53c26d74b8c80e58235551d72a9 (127.24.186.194:38285): tablet 61fedeef49f246d388c31eeae513fc7c (table test-table [id=3ff1acd1262e49a18ee13375131bb54f]) successfully deleted
I20260928 20:24:31.735380 26426 catalog_manager.cc:4841] TS fd595e996fc44805ab40dffe424e90bc (127.24.186.193:38133): tablet 61fedeef49f246d388c31eeae513fc7c (table test-table [id=3ff1acd1262e49a18ee13375131bb54f]) successfully deleted
I20260928 20:24:31.735548 27018 tablet_replica.cc:330] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: stopping tablet replica
I20260928 20:24:31.735620 27018 raft_consensus.cc:2229] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:31.735739 27018 raft_consensus.cc:2258] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:31.736115 27018 ts_tablet_manager.cc:1905] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:31.737850 27018 ts_tablet_manager.cc:1918] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:31.737914 27018 log.cc:1196] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/61fedeef49f246d388c31eeae513fc7c
I20260928 20:24:31.738154 27018 ts_tablet_manager.cc:1939] T 61fedeef49f246d388c31eeae513fc7c P 3b7fb52fd4564ce1ac921da3ebdf3dd4: Deleting consensus metadata
I20260928 20:24:31.738708 26424 catalog_manager.cc:4841] TS 3b7fb52fd4564ce1ac921da3ebdf3dd4 (127.24.186.195:36451): tablet 61fedeef49f246d388c31eeae513fc7c (table test-table [id=3ff1acd1262e49a18ee13375131bb54f]) successfully deleted
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[26395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:31.754118 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.735502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.24.186.193:38133)
0928 20:24:31.735543 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.735545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.736094 (+ 549us) server_negotiation.cc:407] Connection header received
0928 20:24:31.736194 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.736195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.736206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.736236 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.736979 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.737771 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.739345 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.739641 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.739683 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.748756 (+ 9073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.748770 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.748770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.748780 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.752700 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.753180 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.753185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.753185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.753255 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.753477 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.753478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.753479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.753661 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.753826 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.753900 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.753989 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:24:31.754503 26637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36568
I20260928 20:24:31.763566 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:31.755550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.24.186.254:40279)
0928 20:24:31.755595 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:31.755597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:31.755701 (+ 104us) server_negotiation.cc:407] Connection header received
0928 20:24:31.755829 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:31.755830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:31.755847 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:31.755878 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:31.756625 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.757457 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.758543 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:31.758867 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:31.758911 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:31.760560 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:31.760563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:31.760563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:31.760570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:31.762290 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.762691 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.762692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.762693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.762748 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:31.762941 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:31.762941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:31.762942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:31.763072 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:31.763148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:31.763370 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 20:24:31.763465 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:24:31.764217 26440 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54584
I20260928 20:24:31.765118 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26512
I20260928 20:24:31.771732 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26652
I20260928 20:24:31.779258 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26792
I20260928 20:24:31.784935 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 26408
2026-09-28T20:24:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2684 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[27026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[27026](info): set up 2 sockets
Sep 28 20:24:31 dist-test-slave-k477 krb5kdc[27026](info): commencing operation
krb5kdc: starting...
W20260928 20:24:32.848385 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:32 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:32.966039 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41653
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:33.046360 27039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:33.046578 27039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:33.046600 27039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:33.047762 27039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:33.047822 27039 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:24:33.047837 27039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:33.047883 27039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:33.047901 27039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:33.047930 27039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:33.049401 27039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41653
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:33.049662 27039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:33.049898 27039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:33.054639 27044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.054632 27045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.054610 27047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.054889 27039 server_base.cc:1034] running on GCE node
I20260928 20:24:33.055161 27039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:33.055450 27039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:33.056602 27039 hybrid_clock.cc:648] HybridClock initialized: now 1790627073056530 us; error 102 us; skew 500 ppm
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:33.058930 27039 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:33.059346 27039 webserver.cc:460] Webserver started at http://127.24.186.254:38343/ using document root <none> and password file <none>
I20260928 20:24:33.059592 27039 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:33.059655 27039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:33.059837 27039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:33.060902 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "ae5799c811b94c6cad1605ef8511c87a"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "4b11a265a29911834c6f6bd5b3c0309c"
server_key_iv: "975b394912a665b1c8aeb1d7303397ff"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.061250 27039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "ae5799c811b94c6cad1605ef8511c87a"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "4b11a265a29911834c6f6bd5b3c0309c"
server_key_iv: "975b394912a665b1c8aeb1d7303397ff"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.062557 27039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:24:33.063235 27054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.063405 27039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:33.063501 27039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "ae5799c811b94c6cad1605ef8511c87a"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "4b11a265a29911834c6f6bd5b3c0309c"
server_key_iv: "975b394912a665b1c8aeb1d7303397ff"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.063571 27039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:33.076475 27039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:33.078467 27039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:33.078649 27039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:33.083302 27039 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41653
I20260928 20:24:33.083384 27116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41653 every 8 connection(s)
I20260928 20:24:33.083716 27039 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:33.084427 27117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.086637 27117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Bootstrap starting.
I20260928 20:24:33.087221 27117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.087507 27117 log.cc:824] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:33.088251 27117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: No bootstrap required, opened a new log
I20260928 20:24:33.090378 27117 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } }
I20260928 20:24:33.090510 27117 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.090533 27117 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5799c811b94c6cad1605ef8511c87a, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.090698 27117 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } }
I20260928 20:24:33.090764 27117 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:33.090822 27117 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:33.090871 27117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.090982 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27039
I20260928 20:24:33.091069 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:33.091243 25323 external_mini_cluster.cc:1428] Setting key 613b884f88b33ba9664541ff99ea1ab6
I20260928 20:24:33.091528 27117 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } }
I20260928 20:24:33.091897 27117 leader_election.cc:302] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5799c811b94c6cad1605ef8511c87a; no voters:
I20260928 20:24:33.092058 27117 leader_election.cc:288] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:33.092105 27122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:33.092319 27122 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [term 1 LEADER]: Becoming Leader. State: Replica: ae5799c811b94c6cad1605ef8511c87a, State: Running, Role: LEADER
I20260928 20:24:33.092355 27117 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:33.093106 27122 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } }
I20260928 20:24:33.093791 27124 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5799c811b94c6cad1605ef8511c87a. Latest consensus state: current_term: 1 leader_uuid: "ae5799c811b94c6cad1605ef8511c87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } } }
I20260928 20:24:33.093956 27124 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:33.093930 27123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5799c811b94c6cad1605ef8511c87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5799c811b94c6cad1605ef8511c87a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41653 } } }
I20260928 20:24:33.094329 27123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:33.094353 27138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:33.094805 27138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:33.096393 27138 catalog_manager.cc:1347] Generated new cluster ID: 81f645b1baea484d9b66b019555e6c8c
I20260928 20:24:33.096487 27138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.102519 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.091691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.24.186.254:41653)
0928 20:24:33.091863 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.091872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.091904 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:24:33.092023 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.092031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.092107 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.092321 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.093171 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.094677 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.095681 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.096311 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.096422 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.098702 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.098706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.098707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.098720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.100897 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.101295 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.101298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.101298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.101406 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.101618 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.101619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.101619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.101887 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.102064 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.102156 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.102306 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 20:24:33.109139 27138 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:33.109875 27138 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:33.122627 27138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Generated new TSK 0
I20260928 20:24:33.122870 27138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:33.149055 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41653
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:33.242805 27144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:33.243036 27144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:33.243064 27144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:33.244449 27144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:33.244498 27144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:33.244520 27144 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:24:33.244532 27144 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:33.244550 27144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:33.247465 27144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:33.247800 27144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:33.248109 27144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:33.253674 27149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.253674 27152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.253953 27144 server_base.cc:1034] running on GCE node
W20260928 20:24:33.253684 27150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.254380 27144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:33.254673 27144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:33.255923 27144 hybrid_clock.cc:648] HybridClock initialized: now 1790627073255909 us; error 52 us; skew 500 ppm
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:33.258883 27144 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:33.259369 27144 webserver.cc:460] Webserver started at http://127.24.186.193:44331/ using document root <none> and password file <none>
I20260928 20:24:33.259624 27144 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:33.259696 27144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:33.259888 27144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:33.260978 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "f23d6eb15f9f4a719b64116433aa8046"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "73bdd16e78aa7d409f87a285746b30fd"
server_key_iv: "92500ced84b01a42d1857587b54168f3"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.261354 27144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "f23d6eb15f9f4a719b64116433aa8046"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "73bdd16e78aa7d409f87a285746b30fd"
server_key_iv: "92500ced84b01a42d1857587b54168f3"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.262825 27144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:33.263638 27159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.263862 27144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:33.263963 27144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "f23d6eb15f9f4a719b64116433aa8046"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "73bdd16e78aa7d409f87a285746b30fd"
server_key_iv: "92500ced84b01a42d1857587b54168f3"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.264050 27144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:33.278923 27144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:33.280974 27144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:33.281134 27144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:33.281653 27144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:33.281714 27144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.281774 27144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:33.281804 27144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.288671 27144 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:39837
I20260928 20:24:33.288766 27274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:39837 every 8 connection(s)
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.289137 27144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:33.292505 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.282258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:42827 (local address 127.24.186.254:41653)
0928 20:24:33.282307 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.282310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.282519 (+ 209us) server_negotiation.cc:407] Connection header received
0928 20:24:33.283080 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.283081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.283128 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.283201 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.284088 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.284925 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.286539 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.286884 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.286976 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.289726 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.289728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.289728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.289735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.291034 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.291447 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.291448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.291448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.291511 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.291784 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.291786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.291786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.291968 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.292030 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.292314 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.292437 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:24:33.292640 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.282213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.193:42827)
0928 20:24:33.282401 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.282438 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.282965 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.283281 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.283286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.283709 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.283990 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.283995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.285033 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.285034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.286411 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.286414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.286981 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.286982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.287087 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.287838 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.287856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.289632 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.291114 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.291123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.291126 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.291359 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.291594 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.291596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.291596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.291701 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.292031 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.292036 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.292162 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.292255 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.292418 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260928 20:24:33.293419 27275 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41653
I20260928 20:24:33.293538 27275 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:33.293749 27275 heartbeater.cc:511] Master 127.24.186.254:41653 requested a full tablet report, sending...
I20260928 20:24:33.294730 27070 ts_manager.cc:194] Registered new tserver with Master: f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837)
I20260928 20:24:33.294775 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27144
I20260928 20:24:33.294878 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:33.295240 25323 external_mini_cluster.cc:1428] Setting key 5997fb445280576ab5ad88af5e411ad7
I20260928 20:24:33.296007 27070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:42827
I20260928 20:24:33.301578 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.294789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.193:36815)
0928 20:24:33.294922 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.294933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.294993 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.295151 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.295152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.295334 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.295392 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.295394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.296215 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.296216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.296726 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.296727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.297114 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.297115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.297200 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.297681 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.297688 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.298189 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.300124 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.300126 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.300126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.300380 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.300631 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.300632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.300633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.300715 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.301269 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.301270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.301318 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.301382 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.301490 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260928 20:24:33.301674 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.294935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36815 (local address 127.24.186.254:41653)
0928 20:24:33.294983 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.294985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.294994 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:33.295060 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.295061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.295071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.295126 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.295509 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.296080 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.296811 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.297024 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.297114 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.298266 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.298267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.298268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.298275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.300015 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.300478 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.300479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.300480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.300543 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.300976 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.300979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.300979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.301156 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.301258 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.301433 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.301534 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:33.323053 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41653
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:33.412588 27284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:33.412819 27284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:33.412848 27284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:33.414026 27284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:33.414067 27284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:33.414088 27284 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:24:33.414108 27284 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:33.414124 27284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:33.415572 27284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:33.415784 27284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:33.416052 27284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:33.421236 27290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.421271 27292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.421352 27284 server_base.cc:1034] running on GCE node
W20260928 20:24:33.421257 27289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.421696 27284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:33.421922 27284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:33.423084 27284 hybrid_clock.cc:648] HybridClock initialized: now 1790627073423060 us; error 38 us; skew 500 ppm
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:33.425766 27284 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:33.426275 27284 webserver.cc:460] Webserver started at http://127.24.186.194:34981/ using document root <none> and password file <none>
I20260928 20:24:33.426476 27284 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:33.426515 27284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:33.426606 27284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:33.427630 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "d4e96f3e9f564d3fb0fbd889070d7155"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "3b7347d12b3df59169d6e101167a78fe"
server_key_iv: "31ca4af16f420eef7ce4197d17038ed9"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.428085 27284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "d4e96f3e9f564d3fb0fbd889070d7155"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "3b7347d12b3df59169d6e101167a78fe"
server_key_iv: "31ca4af16f420eef7ce4197d17038ed9"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.429416 27284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:33.430083 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.430289 27284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:33.430389 27284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "d4e96f3e9f564d3fb0fbd889070d7155"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "3b7347d12b3df59169d6e101167a78fe"
server_key_iv: "31ca4af16f420eef7ce4197d17038ed9"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.430461 27284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:33.454201 27284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:33.456837 27284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:33.457021 27284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:33.457607 27284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:33.457664 27284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.457705 27284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:33.457731 27284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.465167 27284 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:39501
I20260928 20:24:33.465247 27414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:39501 every 8 connection(s)
I20260928 20:24:33.465726 27284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:33.467963 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.458336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:56511 (local address 127.24.186.254:41653)
0928 20:24:33.458382 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.458384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.458630 (+ 246us) server_negotiation.cc:407] Connection header received
0928 20:24:33.459190 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.459192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.459206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.459286 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.460145 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.461003 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.462068 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.462441 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.462525 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.464993 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.464995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.464996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.465005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.466586 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.467098 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.467099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.467100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.467162 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.467391 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.467392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.467392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.467499 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.467547 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.467770 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.467892 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:24:33.468252 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.458310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.194:56511)
0928 20:24:33.458483 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.458525 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.459068 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.459388 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.459392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.459823 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.460050 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.460055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.461101 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.461102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.461995 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.461998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.462546 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.462548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.462707 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.463276 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.463285 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.464887 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.466687 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.466693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.466695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.466993 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.467235 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.467236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.467236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.467343 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.467550 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.467554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.467670 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.467751 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.467946 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":117,"threads_started":1}
I20260928 20:24:33.468734 27415 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41653
I20260928 20:24:33.468741 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27284
I20260928 20:24:33.468820 27415 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:33.468858 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:33.469014 27415 heartbeater.cc:511] Master 127.24.186.254:41653 requested a full tablet report, sending...
I20260928 20:24:33.469038 25323 external_mini_cluster.cc:1428] Setting key 11596dfb0117dfbb43fccb2b3c5052d4
I20260928 20:24:33.469375 27071 ts_manager.cc:194] Registered new tserver with Master: d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501)
I20260928 20:24:33.469915 27071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:56511
I20260928 20:24:33.475597 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.469794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:38347 (local address 127.24.186.254:41653)
0928 20:24:33.469838 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.469840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.469985 (+ 145us) server_negotiation.cc:407] Connection header received
0928 20:24:33.470119 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.470119 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.470131 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.470159 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.470620 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.471090 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.471793 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.472025 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.472104 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.473052 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.473053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.473054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.473061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.474434 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.474781 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.474783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.474783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.474843 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.475032 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.475033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.475034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.475187 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.475264 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.475417 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.475496 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:24:33.475602 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.469753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.194:38347)
0928 20:24:33.469907 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.469919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.469982 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.470248 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.470249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.470466 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.470537 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.470539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.471186 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.471187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.471694 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.471695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.472107 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.472107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.472186 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.472679 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.472683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.472967 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.474533 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.474534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.474535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.474710 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.474906 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.474906 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.474907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.474967 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.475258 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.475259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.475308 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.475393 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.475497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:33.495400 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41653
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:33.579218 27424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:33.579505 27424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:33.579551 27424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:33.580893 27424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:33.580974 27424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:33.581012 27424 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 20:24:33.581043 27424 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:33.581080 27424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:33.582645 27424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:41653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:33.582930 27424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:33.583175 27424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:33.588223 27430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.588227 27429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:33.588352 27432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:33.588434 27424 server_base.cc:1034] running on GCE node
I20260928 20:24:33.588711 27424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:33.588968 27424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:33.590130 27424 hybrid_clock.cc:648] HybridClock initialized: now 1790627073590117 us; error 35 us; skew 500 ppm
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:33.592988 27424 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:33.593564 27424 webserver.cc:460] Webserver started at http://127.24.186.195:37047/ using document root <none> and password file <none>
I20260928 20:24:33.593847 27424 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:33.593930 27424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:33.594072 27424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:33.595168 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "d605bf622b2643268c6d3c359b7c5aa8"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "92af7da0e91724f6938e07ce0bdea537"
server_key_iv: "6a1b3783df66b313967d4bbb13c49891"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.595486 27424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "d605bf622b2643268c6d3c359b7c5aa8"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "92af7da0e91724f6938e07ce0bdea537"
server_key_iv: "6a1b3783df66b313967d4bbb13c49891"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.596889 27424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:33.597739 27439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.597934 27424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:24:33.598021 27424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "d605bf622b2643268c6d3c359b7c5aa8"
format_stamp: "Formatted at 2026-09-28 20:24:33 on dist-test-slave-k477"
server_key: "92af7da0e91724f6938e07ce0bdea537"
server_key_iv: "6a1b3783df66b313967d4bbb13c49891"
server_key_version: "encryptionkey@0"
I20260928 20:24:33.598083 27424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:33.610090 27424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:33.612824 27424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:33.612953 27424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:33.613433 27424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:33.613469 27424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.613513 27424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:33.613552 27424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.620728 27424 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:44257
I20260928 20:24:33.620826 27554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:44257 every 8 connection(s)
I20260928 20:24:33.621120 27424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:33.624774 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.614057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:59309 (local address 127.24.186.254:41653)
0928 20:24:33.614141 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.614144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.614340 (+ 196us) server_negotiation.cc:407] Connection header received
0928 20:24:33.614865 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.614866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.614881 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.614963 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.615868 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.616719 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.617923 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.618250 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.618337 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.621157 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.621159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.621159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.621168 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.622585 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.623127 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.623128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.623128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.623183 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.623584 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.623585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.623585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.623729 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.623870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.624544 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.624672 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:24:33.624958 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.614033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.195:59309)
0928 20:24:33.614207 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.614248 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.614750 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.615049 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.615054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.615443 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.615743 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.615749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.616843 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.616844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.617823 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.617825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.618369 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.618370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.618552 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.619109 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.619120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.621044 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.622744 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.622751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.622754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.623043 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.623382 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.623382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.623383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.623499 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.624017 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.624036 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.624264 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.624445 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.624671 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260928 20:24:33.626184 27555 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41653
I20260928 20:24:33.626330 27555 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:33.626783 27555 heartbeater.cc:511] Master 127.24.186.254:41653 requested a full tablet report, sending...
I20260928 20:24:33.627565 27070 ts_manager.cc:194] Registered new tserver with Master: d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:33.628153 27070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:59309
I20260928 20:24:33.630751 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27424
I20260928 20:24:33.630867 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:33.631109 25323 external_mini_cluster.cc:1428] Setting key b885578ac33d0edcb9a42de421f48f1d
I20260928 20:24:33.632400 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:24:33.636175 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.627782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.24.186.195:35219)
0928 20:24:33.628006 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.628017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.628187 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.628408 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.628409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.628701 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.628786 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.628788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.629619 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.629620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.630217 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.630218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.630729 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.630732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.630831 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.631512 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.631519 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.632142 (+ 623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.634540 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.634541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.634542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.634864 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.635213 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.635214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.635215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.635311 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.635606 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.635608 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.635692 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.635785 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.636007 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":159,"threads_started":1}
I20260928 20:24:33.636194 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.627864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:35219 (local address 127.24.186.254:41653)
0928 20:24:33.627914 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.627916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.628167 (+ 251us) server_negotiation.cc:407] Connection header received
0928 20:24:33.628305 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.628307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.628323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.628403 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.628892 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.629502 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.630313 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.630603 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.630692 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.632308 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.632310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.632310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.632322 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.634443 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.635002 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.635003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.635004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.635110 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.635397 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.635397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.635398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.635537 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.635607 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.635997 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.636114 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.658906 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.651910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.24.186.254:41653)
0928 20:24:33.651988 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.651990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.652001 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:33.652050 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.652051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.652061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.652117 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.652707 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.653248 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.653987 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.654177 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.654244 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.656598 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.656599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.656599 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.656606 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.657959 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.658290 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.658291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.658291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.658341 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.658491 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.658491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.658492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.658596 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.658668 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.658754 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.658815 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:24:33.660786 27070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:33.662447 27070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:33.669898 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.24.186.194:39501)
0928 20:24:33.665946 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.665950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.665959 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:33.665998 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.666000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.666015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.666122 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.666316 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667428 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.668560 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.669354 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.669436 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.669498 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.669737 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.669785 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":81,"threads_started":1}
I20260928 20:24:33.669883 27573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.0.0.1:55140)
0928 20:24:33.665682 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.665691 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.665728 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.666185 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.666187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.666189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.666231 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.666233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667490 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.667491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.668475 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.668478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.669536 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.669537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.669618 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.669630 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.669720 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.669782 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":77,"threads_started":1}
I20260928 20:24:33.670157 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.24.186.195:44257)
0928 20:24:33.665648 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.665653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.665665 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:33.665707 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.665714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.665728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.665832 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.666041 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667008 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.668594 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.669359 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.669439 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.669498 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.669779 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.670030 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260928 20:24:33.670691 27574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44257 (local address 127.0.0.1:55410)
0928 20:24:33.665465 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.665507 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.665581 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.665845 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.665848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.665850 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.665939 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.665942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667109 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.667110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.668486 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.668489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.669588 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.669590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.669665 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.669669 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.670545 (+ 876us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.670617 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":51,"thread_start_us":69,"threads_started":1}
I20260928 20:24:33.670774 27349 tablet_service.cc:1467] Processing CreateTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 (DEFAULT_TABLE table=test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.671095 27572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39837 (local address 127.0.0.1:57900)
0928 20:24:33.665671 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.665681 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.665728 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.666045 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.666046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.666048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.666105 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.666107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667408 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.667409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.668662 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.668664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.670877 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.670878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.670954 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.670958 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.671009 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.671051 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":40,"thread_start_us":87,"threads_started":1}
I20260928 20:24:33.671180 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d617dcd16544e9b3c932de7c3bc755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.671348 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.665486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.24.186.193:39837)
0928 20:24:33.665845 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.665849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.665859 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:33.665898 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.665900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.665915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.665985 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.666177 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.667308 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.668767 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.670704 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.670782 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.670841 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.671217 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.671267 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":100,"threads_started":1}
I20260928 20:24:33.671341 27489 tablet_service.cc:1467] Processing CreateTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 (DEFAULT_TABLE table=test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.671653 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d617dcd16544e9b3c932de7c3bc755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.671908 27209 tablet_service.cc:1467] Processing CreateTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 (DEFAULT_TABLE table=test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.672170 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10d617dcd16544e9b3c932de7c3bc755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.674940 27578 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Bootstrap starting.
I20260928 20:24:33.675345 27579 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Bootstrap starting.
I20260928 20:24:33.675585 27578 tablet_bootstrap.cc:654] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.675949 27578 log.cc:824] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:33.676281 27580 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Bootstrap starting.
I20260928 20:24:33.676433 27579 tablet_bootstrap.cc:654] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.676759 27579 log.cc:824] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:33.676779 27578 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: No bootstrap required, opened a new log
I20260928 20:24:33.676849 27578 ts_tablet_manager.cc:1397] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:33.676862 27580 tablet_bootstrap.cc:654] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.677176 27580 log.cc:824] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:33.677565 27579 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: No bootstrap required, opened a new log
I20260928 20:24:33.677636 27579 ts_tablet_manager.cc:1397] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:24:33.677867 27580 tablet_bootstrap.cc:492] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: No bootstrap required, opened a new log
I20260928 20:24:33.678026 27580 ts_tablet_manager.cc:1397] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:33.678925 27578 raft_consensus.cc:348] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.679106 27578 raft_consensus.cc:374] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.679145 27578 raft_consensus.cc:729] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.679289 27578 consensus_queue.cc:260] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.679562 27578 ts_tablet_manager.cc:1428] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:33.679687 27275 heartbeater.cc:503] Master 127.24.186.254:41653 was elected leader, sending a full tablet report...
I20260928 20:24:33.679666 27579 raft_consensus.cc:348] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.679839 27579 raft_consensus.cc:374] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.679881 27579 raft_consensus.cc:729] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.679786 27580 raft_consensus.cc:348] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.679991 27580 raft_consensus.cc:374] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.679991 27579 consensus_queue.cc:260] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.680038 27580 raft_consensus.cc:729] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.680167 27580 consensus_queue.cc:260] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.680270 27579 ts_tablet_manager.cc:1428] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:24:33.680394 27555 heartbeater.cc:503] Master 127.24.186.254:41653 was elected leader, sending a full tablet report...
I20260928 20:24:33.680413 27580 ts_tablet_manager.cc:1428] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:24:33.680534 27415 heartbeater.cc:503] Master 127.24.186.254:41653 was elected leader, sending a full tablet report...
I20260928 20:24:33.693312 27585 raft_consensus.cc:482] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:33.693459 27585 raft_consensus.cc:504] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.693887 27585 leader_election.cc:288] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501)
I20260928 20:24:33.697750 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.694157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:59569 (local address 127.24.186.193:39837)
0928 20:24:33.694200 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.694202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.694213 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:33.694243 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.694244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.694255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.694336 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.694544 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.695323 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.696778 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.697366 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.697432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.697441 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.697634 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.697677 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:33.697750 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.694179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:48161 (local address 127.24.186.194:39501)
0928 20:24:33.694223 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.694225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.694463 (+ 238us) server_negotiation.cc:407] Connection header received
0928 20:24:33.694566 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.694567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.694578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.694639 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.694748 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.695412 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.696778 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.697347 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.697432 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.697440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.697634 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.697685 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:24:33.697747 27448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.694033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39837 (local address 127.24.186.195:59569)
0928 20:24:33.694097 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.694110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.694160 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.694393 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.694394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.694396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.694468 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.694469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.695407 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.695408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.696678 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.696681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.697447 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.697448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.697538 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.697541 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.697613 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.697643 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":6}
I20260928 20:24:33.697752 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.694053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.24.186.195:48161)
0928 20:24:33.694394 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.694409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.694472 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.694642 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.694642 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.694644 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.694700 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.694701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.695481 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.695481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.696678 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.696681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.697447 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.697448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.697532 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.697537 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.697601 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.697643 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":188,"thread_start_us":117,"threads_started":1}
I20260928 20:24:33.698143 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d617dcd16544e9b3c932de7c3bc755" candidate_uuid: "d605bf622b2643268c6d3c359b7c5aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046" is_pre_election: true
I20260928 20:24:33.698167 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d617dcd16544e9b3c932de7c3bc755" candidate_uuid: "d605bf622b2643268c6d3c359b7c5aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" is_pre_election: true
I20260928 20:24:33.698326 27369 raft_consensus.cc:2454] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d605bf622b2643268c6d3c359b7c5aa8 in term 0.
I20260928 20:24:33.698326 27229 raft_consensus.cc:2454] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d605bf622b2643268c6d3c359b7c5aa8 in term 0.
I20260928 20:24:33.698563 27441 leader_election.cc:302] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, d605bf622b2643268c6d3c359b7c5aa8; no voters:
I20260928 20:24:33.698774 27585 raft_consensus.cc:2781] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:33.698830 27585 raft_consensus.cc:482] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:33.698860 27585 raft_consensus.cc:3037] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.699643 27585 raft_consensus.cc:504] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.699930 27585 leader_election.cc:288] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [CANDIDATE]: Term 1 election: Requested vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501)
I20260928 20:24:33.700024 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d617dcd16544e9b3c932de7c3bc755" candidate_uuid: "d605bf622b2643268c6d3c359b7c5aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046"
I20260928 20:24:33.700011 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d617dcd16544e9b3c932de7c3bc755" candidate_uuid: "d605bf622b2643268c6d3c359b7c5aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e96f3e9f564d3fb0fbd889070d7155"
I20260928 20:24:33.700102 27369 raft_consensus.cc:3037] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.700102 27229 raft_consensus.cc:3037] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.701031 27229 raft_consensus.cc:2454] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d605bf622b2643268c6d3c359b7c5aa8 in term 1.
I20260928 20:24:33.701051 27369 raft_consensus.cc:2454] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d605bf622b2643268c6d3c359b7c5aa8 in term 1.
I20260928 20:24:33.701215 27441 leader_election.cc:302] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, d605bf622b2643268c6d3c359b7c5aa8; no voters:
I20260928 20:24:33.701306 27585 raft_consensus.cc:2781] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:33.701471 27585 raft_consensus.cc:686] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 LEADER]: Becoming Leader. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Running, Role: LEADER
I20260928 20:24:33.701568 27585 consensus_queue.cc:237] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } }
I20260928 20:24:33.702495 27070 catalog_manager.cc:5495] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195). New cstate: current_term: 1 leader_uuid: "d605bf622b2643268c6d3c359b7c5aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } health_report { overall_health: UNKNOWN } } }
W20260928 20:24:33.716432 27416 tablet.cc:2369] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 20:24:33 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627073, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:33.752477 27121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.745063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.24.186.254:41653)
0928 20:24:33.745144 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.745147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.745253 (+ 106us) server_negotiation.cc:407] Connection header received
0928 20:24:33.745355 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.745356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.745370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.745565 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:33.746089 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.746604 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.747298 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.747478 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.747561 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.749710 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:33.749711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:33.749712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:33.749719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:33.751179 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.751602 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.751603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.751604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.751658 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:33.751882 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:33.751883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:33.751883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:33.752004 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:33.752077 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.752232 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.752392 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":52}
I20260928 20:24:33.752477 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.744810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41653 (local address 127.0.0.1:37494)
0928 20:24:33.745133 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.745150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.745285 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.745586 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.745587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.745944 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:33.745993 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.745996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.746694 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.746695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.747214 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.747216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.747562 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.747563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.747615 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.748198 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:33.748203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:33.749590 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:33.751273 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.751282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.751285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.751529 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.751725 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:33.751726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:33.751726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:33.751795 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:33.752067 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:33.752074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:33.752120 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.752198 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.752345 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":17,"thread_start_us":87,"threads_started":1}
I20260928 20:24:33.755036 27070 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:24:33.755264 27070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:33.756886 27209 tablet_service.cc:1467] Processing CreateTablet for tablet 1fedcd7bbe0d49e2aa7da06303c328c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea8cfb13c53e4005a9578630e1ccf795]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:33.756896 27349 tablet_service.cc:1467] Processing CreateTablet for tablet 1fedcd7bbe0d49e2aa7da06303c328c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea8cfb13c53e4005a9578630e1ccf795]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:33.756894 27489 tablet_service.cc:1467] Processing CreateTablet for tablet 1fedcd7bbe0d49e2aa7da06303c328c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea8cfb13c53e4005a9578630e1ccf795]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:33.757038 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fedcd7bbe0d49e2aa7da06303c328c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.757047 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fedcd7bbe0d49e2aa7da06303c328c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.757047 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fedcd7bbe0d49e2aa7da06303c328c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.758693 27580 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: Bootstrap starting.
I20260928 20:24:33.758694 27578 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: Bootstrap starting.
I20260928 20:24:33.759027 27579 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: Bootstrap starting.
I20260928 20:24:33.759413 27578 tablet_bootstrap.cc:654] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.759449 27580 tablet_bootstrap.cc:654] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.759469 27579 tablet_bootstrap.cc:654] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.760511 27580 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: No bootstrap required, opened a new log
I20260928 20:24:33.760571 27580 ts_tablet_manager.cc:1397] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:24:33.760690 27579 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: No bootstrap required, opened a new log
I20260928 20:24:33.760737 27579 ts_tablet_manager.cc:1397] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 20:24:33.760737 27580 raft_consensus.cc:348] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.760826 27580 raft_consensus.cc:374] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.760857 27580 raft_consensus.cc:729] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.760910 27580 consensus_queue.cc:260] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.760900 27579 raft_consensus.cc:348] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.760947 27578 tablet_bootstrap.cc:492] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: No bootstrap required, opened a new log
I20260928 20:24:33.760967 27579 raft_consensus.cc:374] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.760987 27579 raft_consensus.cc:729] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.761003 27578 ts_tablet_manager.cc:1397] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:33.761004 27580 ts_tablet_manager.cc:1428] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:33.761036 27579 consensus_queue.cc:260] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.761129 27579 ts_tablet_manager.cc:1428] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.761158 27578 raft_consensus.cc:348] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.761150 27586 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.761233 27578 raft_consensus.cc:374] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:33.761236 27586 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.761271 27578 raft_consensus.cc:729] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Initialized, Role: FOLLOWER
I20260928 20:24:33.761322 27578 consensus_queue.cc:260] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.761343 27585 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.761407 27578 ts_tablet_manager.cc:1428] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:33.761417 27585 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.761475 27584 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.761559 27584 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 20:24:33.789861 27276 tablet.cc:2369] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:33.872014 27556 tablet.cc:2369] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:33.905601 27586 raft_consensus.cc:482] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:33.905706 27586 raft_consensus.cc:504] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.906037 27586 leader_election.cc:288] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:33.909008 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.906244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39837 (local address 127.24.186.194:40257)
0928 20:24:33.906301 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.906315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.906372 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.906577 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.906578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.906579 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.906653 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.906654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.907361 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.907362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.908182 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.908184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.908724 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.908819 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.908821 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.908886 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.908929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 20:24:33.909008 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.906406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40257 (local address 127.24.186.193:39837)
0928 20:24:33.906464 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.906466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.906473 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:33.906504 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.906505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.906512 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.906574 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.906718 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.907280 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908287 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.908644 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908725 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.908734 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.908895 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.908931 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:24:33.909180 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.906331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:58939 (local address 127.24.186.195:44257)
0928 20:24:33.906379 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.906382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.906677 (+ 295us) server_negotiation.cc:407] Connection header received
0928 20:24:33.906755 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.906755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.906763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.906820 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:33.906973 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.907477 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908457 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.908836 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908918 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.908928 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.909079 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.909113 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:33.909199 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.906244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44257 (local address 127.24.186.194:58939)
0928 20:24:33.906603 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.906619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.906691 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.906837 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.906838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.906840 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:33.906909 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.906911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.907553 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.907554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.908377 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.908379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.908931 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.908932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.909001 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.909004 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.909061 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.909098 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":193,"thread_start_us":119,"threads_started":1}
I20260928 20:24:33.909271 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fedcd7bbe0d49e2aa7da06303c328c5" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046" is_pre_election: true
I20260928 20:24:33.909374 27229 raft_consensus.cc:2454] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:33.909523 27302 leader_election.cc:302] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:33.909518 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fedcd7bbe0d49e2aa7da06303c328c5" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8" is_pre_election: true
I20260928 20:24:33.909621 27509 raft_consensus.cc:2454] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:33.909669 27586 raft_consensus.cc:2781] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:33.909708 27586 raft_consensus.cc:482] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:33.909735 27586 raft_consensus.cc:3037] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.910439 27586 raft_consensus.cc:504] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.910575 27586 leader_election.cc:288] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Requested vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:33.910756 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fedcd7bbe0d49e2aa7da06303c328c5" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8"
I20260928 20:24:33.910770 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fedcd7bbe0d49e2aa7da06303c328c5" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046"
I20260928 20:24:33.910853 27229 raft_consensus.cc:3037] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.910856 27509 raft_consensus.cc:3037] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:33.911428 27229 raft_consensus.cc:2454] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:33.911579 27302 leader_election.cc:302] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:33.911675 27586 raft_consensus.cc:2781] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:33.911708 27509 raft_consensus.cc:2454] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:33.911844 27586 raft_consensus.cc:686] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 LEADER]: Becoming Leader. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Running, Role: LEADER
I20260928 20:24:33.911944 27586 consensus_queue.cc:237] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:33.912348 27600 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: TxnStatusTablet state changed. Reason: New leader d4e96f3e9f564d3fb0fbd889070d7155. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.912423 27600 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:33.912602 27602 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:24:33.912735 27602 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:24:33.912746 27069 catalog_manager.cc:5495] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:33.946273 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.943772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.24.186.194:39501)
0928 20:24:33.943848 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.943850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.943861 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:33.943980 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.943981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.944006 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.944082 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:33.944231 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.944728 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.945466 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.945666 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.945740 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.945932 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.946042 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.946146 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.946167 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.946187 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:24:33.946275 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.943658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.0.0.1:55146)
0928 20:24:33.943771 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.943783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.943884 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.944103 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.944104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.944106 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:33.944166 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.944168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.944816 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.944817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.945384 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.945385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.945739 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.945740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.945809 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.945824 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.946108 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.946118 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.946158 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.946206 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 20:24:33.948132 27229 raft_consensus.cc:1261] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:33.948132 27509 raft_consensus.cc:1261] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:33.948473 27586 consensus_queue.cc:1035] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:33.948617 27600 consensus_queue.cc:1035] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:33.949751 27585 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: TxnStatusTablet state changed. Reason: New leader d4e96f3e9f564d3fb0fbd889070d7155. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.949848 27585 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.949803 27584 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: TxnStatusTablet state changed. Reason: New leader d4e96f3e9f564d3fb0fbd889070d7155. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.949918 27584 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.952240 27586 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.952392 27586 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:33.952374 27607 mvcc.cc:205] Tried to move back new op lower bound from 7334408494888714240 to 7334408494743756800. Current Snapshot: MvccSnapshot[applied={T|T < 7334408494888714240}]
I20260928 20:24:33.952468 27585 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.952535 27585 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d605bf622b2643268c6d3c359b7c5aa8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.953063 27605 mvcc.cc:205] Tried to move back new op lower bound from 7334408494888714240 to 7334408494743756800. Current Snapshot: MvccSnapshot[applied={T|T < 7334408494888714240}]
I20260928 20:24:33.953341 27600 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.953438 27600 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P d4e96f3e9f564d3fb0fbd889070d7155: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:33.953691 27584 tablet_replica.cc:439] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } } }
I20260928 20:24:33.953790 27584 tablet_replica.cc:442] T 1fedcd7bbe0d49e2aa7da06303c328c5 P f23d6eb15f9f4a719b64116433aa8046: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:33.954155 27606 mvcc.cc:205] Tried to move back new op lower bound from 7334408494888714240 to 7334408494743756800. Current Snapshot: MvccSnapshot[applied={T|T < 7334408494888714240 or (T in {7334408494888714240})}]
I20260928 20:24:33.960466 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.957988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.24.186.195:44257)
0928 20:24:33.958067 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.958069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.958078 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:33.958118 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.958119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.958135 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.958214 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:33.958370 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.958850 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.959618 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.959848 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.959927 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.960103 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.960220 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.960329 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.960350 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.960372 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 20:24:33.967257 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.963613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.24.186.195:55349)
0928 20:24:33.963789 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.963819 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.964084 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.964307 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.964308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.964310 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:33.964491 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.964496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.965307 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.965308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.966255 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.966258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.966727 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.966728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.966799 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.966819 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.967049 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.967052 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.967139 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.967188 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 20:24:33.967258 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.963710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:55349 (local address 127.24.186.194:39501)
0928 20:24:33.963769 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.963771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.964080 (+ 309us) server_negotiation.cc:407] Connection header received
0928 20:24:33.964182 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.964183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.964207 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.964273 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:33.964587 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.965211 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.966418 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.966631 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.966689 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.966947 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.966985 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.967036 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.967146 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.967172 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:24:33.970021 27229 raft_consensus.cc:1261] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Refusing update from remote peer d605bf622b2643268c6d3c359b7c5aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:33.970031 27369 raft_consensus.cc:1261] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Refusing update from remote peer d605bf622b2643268c6d3c359b7c5aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:33.970314 27588 consensus_queue.cc:1035] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:33.970479 27585 consensus_queue.cc:1035] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:33.979691 27575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.976814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:33727 (local address 127.24.186.195:44257)
0928 20:24:33.976934 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:33.976936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:33.977001 (+ 65us) server_negotiation.cc:407] Connection header received
0928 20:24:33.977099 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:33.977100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:33.977110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:33.977138 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:33.977323 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.977892 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.978972 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.979260 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.979314 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.979501 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.979529 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.979563 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:33.979612 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 20:24:33.979632 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 20:24:33.979749 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:33.976777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44257 (local address 127.24.186.194:33727)
0928 20:24:33.976942 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:33.976954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:33.977017 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:33.977206 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:33.977207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:33.977208 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:33.977258 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.977260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.977970 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.977971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.978796 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:33.978799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:33.979333 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:33.979334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:33.979400 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:33.979416 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:33.979566 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:33.979568 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:33.979605 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 20:24:33.979663 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 20:24:33.990399 27069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:33.990535 27069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:33.991340 27069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Sending DeleteTablet for 3 replicas of tablet 10d617dcd16544e9b3c932de7c3bc755
I20260928 20:24:33.991677 27489 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55410
I20260928 20:24:33.991843 27209 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57900
I20260928 20:24:33.992076 27625 tablet_replica.cc:330] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: stopping tablet replica
I20260928 20:24:33.992067 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet 10d617dcd16544e9b3c932de7c3bc755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55140
I20260928 20:24:33.992190 27625 raft_consensus.cc:2229] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:33.992261 27625 raft_consensus.cc:2258] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:33.992352 27626 tablet_replica.cc:330] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: stopping tablet replica
I20260928 20:24:33.992527 27069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:33.992677 27069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:33.992628 27626 raft_consensus.cc:2229] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:33.992960 27626 raft_consensus.cc:2258] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:33.993948 27625 ts_tablet_manager.cc:1905] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:33.994120 27349 tablet_service.cc:1467] Processing CreateTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 (DEFAULT_TABLE table=test-table [id=9b2f4b928d944779b8f9284f63c3125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.994232 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d46a3e21174f1eba8ba02c633f93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.994207 27489 tablet_service.cc:1467] Processing CreateTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 (DEFAULT_TABLE table=test-table [id=9b2f4b928d944779b8f9284f63c3125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.994328 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d46a3e21174f1eba8ba02c633f93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.994906 27626 ts_tablet_manager.cc:1905] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:33.995648 27580 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Bootstrap starting.
I20260928 20:24:33.995939 27579 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Bootstrap starting.
I20260928 20:24:33.996397 27580 tablet_bootstrap.cc:654] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.996440 27579 tablet_bootstrap.cc:654] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.996946 27626 ts_tablet_manager.cc:1918] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:33.997018 27626 log.cc:1196] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/10d617dcd16544e9b3c932de7c3bc755
I20260928 20:24:33.997308 27626 ts_tablet_manager.cc:1939] T 10d617dcd16544e9b3c932de7c3bc755 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting consensus metadata
I20260928 20:24:33.997395 27209 tablet_service.cc:1467] Processing CreateTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 (DEFAULT_TABLE table=test-table [id=9b2f4b928d944779b8f9284f63c3125d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:33.997510 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1d46a3e21174f1eba8ba02c633f93a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:33.998651 27625 ts_tablet_manager.cc:1918] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:33.998721 27625 log.cc:1196] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/10d617dcd16544e9b3c932de7c3bc755
I20260928 20:24:33.998904 27578 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Bootstrap starting.
I20260928 20:24:33.998991 27625 ts_tablet_manager.cc:1939] T 10d617dcd16544e9b3c932de7c3bc755 P f23d6eb15f9f4a719b64116433aa8046: Deleting consensus metadata
I20260928 20:24:33.999336 27578 tablet_bootstrap.cc:654] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:33.999907 27624 tablet_replica.cc:330] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: stopping tablet replica
I20260928 20:24:34.000031 27624 raft_consensus.cc:2229] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:34.000036 27578 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: No bootstrap required, opened a new log
I20260928 20:24:34.000079 27578 ts_tablet_manager.cc:1397] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:34.000093 27057 catalog_manager.cc:4841] TS f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837): tablet 10d617dcd16544e9b3c932de7c3bc755 (table test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]) successfully deleted
I20260928 20:24:34.000145 27624 raft_consensus.cc:2258] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:34.000229 27578 raft_consensus.cc:348] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.000321 27578 raft_consensus.cc:374] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:34.000344 27578 raft_consensus.cc:729] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Initialized, Role: FOLLOWER
I20260928 20:24:34.000386 27578 consensus_queue.cc:260] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.000483 27578 ts_tablet_manager.cc:1428] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:34.000964 27056 catalog_manager.cc:4841] TS d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501): tablet 10d617dcd16544e9b3c932de7c3bc755 (table test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]) successfully deleted
I20260928 20:24:34.001353 27579 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: No bootstrap required, opened a new log
I20260928 20:24:34.001407 27579 ts_tablet_manager.cc:1397] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s
I20260928 20:24:34.001571 27579 raft_consensus.cc:348] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.001646 27579 raft_consensus.cc:374] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:34.001667 27579 raft_consensus.cc:729] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Initialized, Role: FOLLOWER
I20260928 20:24:34.001711 27579 consensus_queue.cc:260] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.001801 27579 ts_tablet_manager.cc:1428] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:34.004120 27624 ts_tablet_manager.cc:1905] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:34.007031 27624 ts_tablet_manager.cc:1918] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:34.007206 27624 log.cc:1196] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/10d617dcd16544e9b3c932de7c3bc755
I20260928 20:24:34.007476 27580 tablet_bootstrap.cc:492] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: No bootstrap required, opened a new log
I20260928 20:24:34.007544 27580 ts_tablet_manager.cc:1397] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260928 20:24:34.007685 27624 ts_tablet_manager.cc:1939] T 10d617dcd16544e9b3c932de7c3bc755 P d605bf622b2643268c6d3c359b7c5aa8: Deleting consensus metadata
I20260928 20:24:34.007727 27580 raft_consensus.cc:348] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.007833 27580 raft_consensus.cc:374] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:34.007861 27580 raft_consensus.cc:729] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Initialized, Role: FOLLOWER
I20260928 20:24:34.007905 27580 consensus_queue.cc:260] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.007995 27580 ts_tablet_manager.cc:1428] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:34.008401 27055 catalog_manager.cc:4841] TS d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257): tablet 10d617dcd16544e9b3c932de7c3bc755 (table test-table [id=be8e3a4a16924b1eab98ae3e6a4ea3f6]) successfully deleted
I20260928 20:24:34.204882 27600 raft_consensus.cc:482] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:34.205010 27600 raft_consensus.cc:504] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.205205 27600 leader_election.cc:288] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:34.205479 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d46a3e21174f1eba8ba02c633f93a4" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046" is_pre_election: true
I20260928 20:24:34.205587 27229 raft_consensus.cc:2454] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:34.205646 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d46a3e21174f1eba8ba02c633f93a4" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8" is_pre_election: true
I20260928 20:24:34.205848 27509 raft_consensus.cc:2454] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:34.205848 27302 leader_election.cc:302] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:34.205960 27600 raft_consensus.cc:2781] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:34.206012 27600 raft_consensus.cc:482] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:34.206043 27600 raft_consensus.cc:3037] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:34.206782 27600 raft_consensus.cc:504] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.206941 27600 leader_election.cc:288] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Requested vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:34.207075 27229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d46a3e21174f1eba8ba02c633f93a4" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046"
I20260928 20:24:34.207142 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1d46a3e21174f1eba8ba02c633f93a4" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8"
I20260928 20:24:34.207188 27229 raft_consensus.cc:3037] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:34.207201 27509 raft_consensus.cc:3037] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:34.208024 27509 raft_consensus.cc:2454] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:34.208024 27229 raft_consensus.cc:2454] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:34.208210 27302 leader_election.cc:302] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:34.208334 27600 raft_consensus.cc:2781] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:34.208400 27600 raft_consensus.cc:686] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 LEADER]: Becoming Leader. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Running, Role: LEADER
I20260928 20:24:34.208454 27600 consensus_queue.cc:237] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:34.209079 27069 catalog_manager.cc:5495] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:34.233877 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:34.231419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.24.186.194:39501)
0928 20:24:34.231463 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:34.231465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:34.231489 (+ 24us) server_negotiation.cc:407] Connection header received
0928 20:24:34.231558 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:34.231559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:34.231573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:34.231623 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:34.231731 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.232229 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.233115 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.233351 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.233435 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:34.233571 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:34.233606 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:34.233704 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:34.233764 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 20:24:34.233787 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:34.238245 27577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:34.236238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:37769 (local address 127.24.186.194:39501)
0928 20:24:34.236272 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:34.236274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:34.236315 (+ 41us) server_negotiation.cc:407] Connection header received
0928 20:24:34.236376 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:34.236376 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:34.236383 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:34.236427 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:34.236531 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.236961 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.237659 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.237828 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.237877 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:34.238019 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:34.238042 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:34.238079 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:34.238143 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 20:24:34.238156 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:34.238247 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:34.236189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.24.186.194:37769)
0928 20:24:34.236262 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:34.236275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:34.236337 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:34.236444 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:34.236444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:34.236445 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:34.236481 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:34.236482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.237044 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.237045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:34.237580 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:34.237582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:34.237907 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:34.237908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:34.237952 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:34.237959 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:34.238091 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:34.238093 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:34.238137 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 20:24:34.238175 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 20:24:34.240228 27229 raft_consensus.cc:1261] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:34.240232 27509 raft_consensus.cc:1261] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:34.240448 27586 consensus_queue.cc:1035] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:34.240561 27600 consensus_queue.cc:1035] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 20:24:34.245126 27605 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 20:24:35.767241 27071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:35.767347 27071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:35.768225 27071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Sending DeleteTablet for 3 replicas of tablet a1d46a3e21174f1eba8ba02c633f93a4
I20260928 20:24:35.768535 27209 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57900
I20260928 20:24:35.768558 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55140
I20260928 20:24:35.768723 27489 tablet_service.cc:1514] Processing DeleteTablet for tablet a1d46a3e21174f1eba8ba02c633f93a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55410
I20260928 20:24:35.768899 27656 tablet_replica.cc:330] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: stopping tablet replica
I20260928 20:24:35.768949 27656 raft_consensus.cc:2229] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:35.768985 27656 raft_consensus.cc:2258] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.769219 27656 ts_tablet_manager.cc:1905] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.770377 27654 tablet_replica.cc:330] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: stopping tablet replica
I20260928 20:24:35.770452 27654 raft_consensus.cc:2229] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:35.770520 27654 raft_consensus.cc:2258] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.770824 27654 ts_tablet_manager.cc:1905] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.770669 27655 tablet_replica.cc:330] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: stopping tablet replica
I20260928 20:24:35.771198 27655 raft_consensus.cc:2229] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:35.771306 27655 raft_consensus.cc:2258] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.771509 27655 ts_tablet_manager.cc:1905] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.772771 27654 ts_tablet_manager.cc:1918] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:24:35.772841 27654 log.cc:1196] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/a1d46a3e21174f1eba8ba02c633f93a4
I20260928 20:24:35.773092 27654 ts_tablet_manager.cc:1939] T a1d46a3e21174f1eba8ba02c633f93a4 P f23d6eb15f9f4a719b64116433aa8046: Deleting consensus metadata
I20260928 20:24:35.773547 27655 ts_tablet_manager.cc:1918] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:24:35.773607 27655 log.cc:1196] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/a1d46a3e21174f1eba8ba02c633f93a4
I20260928 20:24:35.773628 27057 catalog_manager.cc:4841] TS f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837): tablet a1d46a3e21174f1eba8ba02c633f93a4 (table test-table [id=9b2f4b928d944779b8f9284f63c3125d]) successfully deleted
I20260928 20:24:35.773869 27655 ts_tablet_manager.cc:1939] T a1d46a3e21174f1eba8ba02c633f93a4 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting consensus metadata
I20260928 20:24:35.773938 27656 ts_tablet_manager.cc:1918] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:24:35.774001 27656 log.cc:1196] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/a1d46a3e21174f1eba8ba02c633f93a4
I20260928 20:24:35.774197 27056 catalog_manager.cc:4841] TS d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501): tablet a1d46a3e21174f1eba8ba02c633f93a4 (table test-table [id=9b2f4b928d944779b8f9284f63c3125d]) successfully deleted
I20260928 20:24:35.774245 27656 ts_tablet_manager.cc:1939] T a1d46a3e21174f1eba8ba02c633f93a4 P d605bf622b2643268c6d3c359b7c5aa8: Deleting consensus metadata
I20260928 20:24:35.775190 27055 catalog_manager.cc:4841] TS d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257): tablet a1d46a3e21174f1eba8ba02c633f93a4 (table test-table [id=9b2f4b928d944779b8f9284f63c3125d]) successfully deleted
Sep 28 20:24:35 dist-test-slave-k477 krb5kdc[27026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:24:35 dist-test-slave-k477 krb5kdc[27026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627075, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:35.794776 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.787538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.24.186.254:41653)
0928 20:24:35.787700 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.787702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.787711 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:35.787776 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.787777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.787788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.787892 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:35.788427 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.788930 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.789609 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.789795 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.789843 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.792116 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:35.792118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:35.792118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:35.792124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:35.793476 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:35.793899 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:35.793901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:35.793901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:35.793967 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:35.794195 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:35.794197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:35.794197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:35.794358 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:35.794449 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.794540 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.794628 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260928 20:24:35.796255 27071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:35.796416 27071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:35.797740 27209 tablet_service.cc:1467] Processing CreateTablet for tablet 969362b8793a4ffe8f528fb8b443475f (DEFAULT_TABLE table=test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.797869 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969362b8793a4ffe8f528fb8b443475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.797973 27349 tablet_service.cc:1467] Processing CreateTablet for tablet 969362b8793a4ffe8f528fb8b443475f (DEFAULT_TABLE table=test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.798295 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969362b8793a4ffe8f528fb8b443475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.799281 27665 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Bootstrap starting.
I20260928 20:24:35.799690 27665 tablet_bootstrap.cc:654] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.800387 27665 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: No bootstrap required, opened a new log
I20260928 20:24:35.800395 27667 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Bootstrap starting.
I20260928 20:24:35.800441 27665 ts_tablet_manager.cc:1397] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:35.800426 27489 tablet_service.cc:1467] Processing CreateTablet for tablet 969362b8793a4ffe8f528fb8b443475f (DEFAULT_TABLE table=test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.800544 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969362b8793a4ffe8f528fb8b443475f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.800624 27665 raft_consensus.cc:348] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.800745 27665 raft_consensus.cc:374] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.800792 27665 raft_consensus.cc:729] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.800802 27667 tablet_bootstrap.cc:654] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.800846 27665 consensus_queue.cc:260] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.801031 27665 ts_tablet_manager.cc:1428] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 20:24:35.801537 27667 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: No bootstrap required, opened a new log
I20260928 20:24:35.801599 27667 ts_tablet_manager.cc:1397] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:35.801735 27667 raft_consensus.cc:348] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.801821 27667 raft_consensus.cc:374] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.801844 27667 raft_consensus.cc:729] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.801892 27667 consensus_queue.cc:260] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.801960 27670 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Bootstrap starting.
I20260928 20:24:35.802117 27667 ts_tablet_manager.cc:1428] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:35.802388 27670 tablet_bootstrap.cc:654] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.803056 27670 tablet_bootstrap.cc:492] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: No bootstrap required, opened a new log
I20260928 20:24:35.803097 27670 ts_tablet_manager.cc:1397] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:35.803179 27670 raft_consensus.cc:348] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.803229 27670 raft_consensus.cc:374] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.803242 27670 raft_consensus.cc:729] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.803296 27670 consensus_queue.cc:260] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.803462 27670 ts_tablet_manager.cc:1428] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:35.817189 27642 raft_consensus.cc:482] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:35.817289 27642 raft_consensus.cc:504] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.817459 27642 leader_election.cc:288] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:35.817663 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969362b8793a4ffe8f528fb8b443475f" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8" is_pre_election: true
I20260928 20:24:35.817711 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969362b8793a4ffe8f528fb8b443475f" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046" is_pre_election: true
I20260928 20:24:35.817775 27509 raft_consensus.cc:2454] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:35.817811 27228 raft_consensus.cc:2454] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 0.
I20260928 20:24:35.817994 27300 leader_election.cc:302] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, d605bf622b2643268c6d3c359b7c5aa8; no voters:
I20260928 20:24:35.818107 27642 raft_consensus.cc:2781] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:35.818176 27642 raft_consensus.cc:482] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:35.818202 27642 raft_consensus.cc:3037] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.818935 27642 raft_consensus.cc:504] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.819084 27642 leader_election.cc:288] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Requested vote from peers f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:35.819240 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969362b8793a4ffe8f528fb8b443475f" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8"
I20260928 20:24:35.819253 27228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969362b8793a4ffe8f528fb8b443475f" candidate_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f23d6eb15f9f4a719b64116433aa8046"
I20260928 20:24:35.819320 27509 raft_consensus.cc:3037] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.819326 27228 raft_consensus.cc:3037] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.820042 27509 raft_consensus.cc:2454] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:35.820067 27228 raft_consensus.cc:2454] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e96f3e9f564d3fb0fbd889070d7155 in term 1.
I20260928 20:24:35.820215 27300 leader_election.cc:302] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4e96f3e9f564d3fb0fbd889070d7155, d605bf622b2643268c6d3c359b7c5aa8; no voters:
I20260928 20:24:35.820325 27642 raft_consensus.cc:2781] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:35.820387 27642 raft_consensus.cc:686] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 LEADER]: Becoming Leader. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Running, Role: LEADER
I20260928 20:24:35.820441 27642 consensus_queue.cc:237] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.821028 27071 catalog_manager.cc:5495] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:35.831598 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.829424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.24.186.194:39501)
0928 20:24:35.829559 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.829560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.829567 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:35.829586 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.829586 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.829595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.829651 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:35.829776 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.830246 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.830942 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.831147 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.831206 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.831330 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.831357 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.831396 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.831448 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.831464 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 20:24:35.834157 27509 raft_consensus.cc:1261] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:35.834386 27228 raft_consensus.cc:1261] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Refusing update from remote peer d4e96f3e9f564d3fb0fbd889070d7155: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:35.834408 27637 consensus_queue.cc:1035] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:35.834613 27637 consensus_queue.cc:1035] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:35.843662 27071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:35.843757 27071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:35.844393 27071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Sending DeleteTablet for 3 replicas of tablet 969362b8793a4ffe8f528fb8b443475f
I20260928 20:24:35.844635 27209 tablet_service.cc:1514] Processing DeleteTablet for tablet 969362b8793a4ffe8f528fb8b443475f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57900
I20260928 20:24:35.844647 27489 tablet_service.cc:1514] Processing DeleteTablet for tablet 969362b8793a4ffe8f528fb8b443475f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55410
I20260928 20:24:35.844642 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet 969362b8793a4ffe8f528fb8b443475f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:35 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55140
I20260928 20:24:35.844726 27654 tablet_replica.cc:330] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: stopping tablet replica
I20260928 20:24:35.844748 27655 tablet_replica.cc:330] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: stopping tablet replica
I20260928 20:24:35.844750 27656 tablet_replica.cc:330] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: stopping tablet replica
I20260928 20:24:35.844766 27654 raft_consensus.cc:2229] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:35.844794 27655 raft_consensus.cc:2229] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:35.844806 27654 raft_consensus.cc:2258] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.844816 27656 raft_consensus.cc:2229] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:35.844846 27655 raft_consensus.cc:2258] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.844853 27656 raft_consensus.cc:2258] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:35.845239 27656 ts_tablet_manager.cc:1905] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.845371 27655 ts_tablet_manager.cc:1905] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.845419 27654 ts_tablet_manager.cc:1905] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:35.847035 27654 ts_tablet_manager.cc:1918] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:35.846973 27071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 20:24:35.847078 27655 ts_tablet_manager.cc:1918] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:35.847078 27656 ts_tablet_manager.cc:1918] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:35.847102 27654 log.cc:1196] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/969362b8793a4ffe8f528fb8b443475f
I20260928 20:24:35.847134 27656 log.cc:1196] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/969362b8793a4ffe8f528fb8b443475f
I20260928 20:24:35.847134 27655 log.cc:1196] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/969362b8793a4ffe8f528fb8b443475f
W20260928 20:24:35.847137 27071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:35.847352 27654 ts_tablet_manager.cc:1939] T 969362b8793a4ffe8f528fb8b443475f P f23d6eb15f9f4a719b64116433aa8046: Deleting consensus metadata
I20260928 20:24:35.847371 27656 ts_tablet_manager.cc:1939] T 969362b8793a4ffe8f528fb8b443475f P d605bf622b2643268c6d3c359b7c5aa8: Deleting consensus metadata
I20260928 20:24:35.847370 27655 ts_tablet_manager.cc:1939] T 969362b8793a4ffe8f528fb8b443475f P d4e96f3e9f564d3fb0fbd889070d7155: Deleting consensus metadata
I20260928 20:24:35.847702 27057 catalog_manager.cc:4841] TS f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837): tablet 969362b8793a4ffe8f528fb8b443475f (table test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]) successfully deleted
I20260928 20:24:35.847725 27056 catalog_manager.cc:4841] TS d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501): tablet 969362b8793a4ffe8f528fb8b443475f (table test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]) successfully deleted
I20260928 20:24:35.847913 27055 catalog_manager.cc:4841] TS d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257): tablet 969362b8793a4ffe8f528fb8b443475f (table test-table [id=a8c10b30f3f846ccaa4f8c2dae08f98c]) successfully deleted
I20260928 20:24:35.848520 27209 tablet_service.cc:1467] Processing CreateTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 (DEFAULT_TABLE table=test-table [id=54b48d1d67f7431abfaf1d569c07afdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.848531 27349 tablet_service.cc:1467] Processing CreateTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 (DEFAULT_TABLE table=test-table [id=54b48d1d67f7431abfaf1d569c07afdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.848562 27489 tablet_service.cc:1467] Processing CreateTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 (DEFAULT_TABLE table=test-table [id=54b48d1d67f7431abfaf1d569c07afdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:35.848639 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41112f4e8c2f43299d84c676447c3ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.848639 27209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41112f4e8c2f43299d84c676447c3ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.848644 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41112f4e8c2f43299d84c676447c3ce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:35.849509 27667 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Bootstrap starting.
I20260928 20:24:35.849821 27670 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Bootstrap starting.
I20260928 20:24:35.849862 27667 tablet_bootstrap.cc:654] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.849821 27665 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046: Bootstrap starting.
I20260928 20:24:35.850184 27670 tablet_bootstrap.cc:654] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.850420 27665 tablet_bootstrap.cc:654] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:35.850567 27667 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: No bootstrap required, opened a new log
I20260928 20:24:35.850623 27667 ts_tablet_manager.cc:1397] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:35.850754 27667 raft_consensus.cc:348] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.850853 27667 raft_consensus.cc:374] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.850885 27667 raft_consensus.cc:729] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e96f3e9f564d3fb0fbd889070d7155, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.850932 27667 consensus_queue.cc:260] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.851033 27667 ts_tablet_manager.cc:1428] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:35.851034 27665 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046: No bootstrap required, opened a new log
I20260928 20:24:35.851078 27665 ts_tablet_manager.cc:1397] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:35.851217 27665 raft_consensus.cc:348] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.851298 27665 raft_consensus.cc:374] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.851332 27665 raft_consensus.cc:729] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.851375 27665 consensus_queue.cc:260] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.851477 27665 ts_tablet_manager.cc:1428] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:35.851511 27670 tablet_bootstrap.cc:492] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: No bootstrap required, opened a new log
I20260928 20:24:35.851553 27670 ts_tablet_manager.cc:1397] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:35.851661 27670 raft_consensus.cc:348] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.851739 27670 raft_consensus.cc:374] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:35.851773 27670 raft_consensus.cc:729] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d605bf622b2643268c6d3c359b7c5aa8, State: Initialized, Role: FOLLOWER
I20260928 20:24:35.851857 27670 consensus_queue.cc:260] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.851958 27670 ts_tablet_manager.cc:1428] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:35.855086 27668 raft_consensus.cc:482] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:35.855180 27668 raft_consensus.cc:504] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.855556 27668 leader_election.cc:288] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:35.858815 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.855860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:35359 (local address 127.24.186.195:44257)
0928 20:24:35.855957 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.855959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.855967 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:35.855989 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.855990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.856000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.856069 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:35.856251 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.856810 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.857801 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.858197 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.858256 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.858263 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.858643 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.858700 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
I20260928 20:24:35.858975 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.855705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.24.186.193:36495)
0928 20:24:35.856023 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:35.856034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:35.856083 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:35.856251 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:35.856251 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:35.856253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:35.856321 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.856322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.856959 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.856959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.857711 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.857713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.858719 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.858720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.858794 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.858797 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:35.858845 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:24:35.858887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":2,"spinlock_wait_cycles":1280,"thread_start_us":139,"threads_started":1}
I20260928 20:24:35.859133 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.855852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36495 (local address 127.24.186.194:39501)
0928 20:24:35.855901 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.855903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.856136 (+ 233us) server_negotiation.cc:407] Connection header received
0928 20:24:35.856164 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.856165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.856187 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.856241 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:35.856386 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.856836 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.857803 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.858593 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.858667 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.858684 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.858873 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.858960 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:24:35.859146 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.855702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44257 (local address 127.24.186.193:35359)
0928 20:24:35.855833 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:35.855846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:35.855899 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:35.856076 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:35.856077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:35.856080 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:35.856161 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.856163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.856889 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.856890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.857713 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.857715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.858335 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.858337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.858519 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.858528 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 20:24:35.858589 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:24:35.858658 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260928 20:24:35.859438 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41112f4e8c2f43299d84c676447c3ce6" candidate_uuid: "f23d6eb15f9f4a719b64116433aa8046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" is_pre_election: true
I20260928 20:24:35.859467 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41112f4e8c2f43299d84c676447c3ce6" candidate_uuid: "f23d6eb15f9f4a719b64116433aa8046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8" is_pre_election: true
I20260928 20:24:35.859534 27509 raft_consensus.cc:2454] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23d6eb15f9f4a719b64116433aa8046 in term 0.
I20260928 20:24:35.859534 27369 raft_consensus.cc:2454] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f23d6eb15f9f4a719b64116433aa8046 in term 0.
I20260928 20:24:35.859759 27160 leader_election.cc:302] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d605bf622b2643268c6d3c359b7c5aa8, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:35.859946 27668 raft_consensus.cc:2781] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:35.859987 27668 raft_consensus.cc:482] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:35.860010 27668 raft_consensus.cc:3037] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.860744 27668 raft_consensus.cc:504] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.860888 27668 leader_election.cc:288] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [CANDIDATE]: Term 1 election: Requested vote from peers d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501), d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257)
I20260928 20:24:35.861034 27509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41112f4e8c2f43299d84c676447c3ce6" candidate_uuid: "f23d6eb15f9f4a719b64116433aa8046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d605bf622b2643268c6d3c359b7c5aa8"
I20260928 20:24:35.861034 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41112f4e8c2f43299d84c676447c3ce6" candidate_uuid: "f23d6eb15f9f4a719b64116433aa8046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e96f3e9f564d3fb0fbd889070d7155"
I20260928 20:24:35.861110 27369 raft_consensus.cc:3037] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.861110 27509 raft_consensus.cc:3037] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:35.862694 27369 raft_consensus.cc:2454] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23d6eb15f9f4a719b64116433aa8046 in term 1.
I20260928 20:24:35.862689 27509 raft_consensus.cc:2454] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f23d6eb15f9f4a719b64116433aa8046 in term 1.
I20260928 20:24:35.863018 27160 leader_election.cc:302] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d605bf622b2643268c6d3c359b7c5aa8, f23d6eb15f9f4a719b64116433aa8046; no voters:
I20260928 20:24:35.863119 27668 raft_consensus.cc:2781] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:35.863291 27668 raft_consensus.cc:686] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [term 1 LEADER]: Becoming Leader. State: Replica: f23d6eb15f9f4a719b64116433aa8046, State: Running, Role: LEADER
I20260928 20:24:35.863350 27668 consensus_queue.cc:237] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } }
I20260928 20:24:35.864064 27069 catalog_manager.cc:5495] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 reported cstate change: term changed from 0 to 1, leader changed from <none> to f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "f23d6eb15f9f4a719b64116433aa8046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f23d6eb15f9f4a719b64116433aa8046" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:35.872467 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.870012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.24.186.193:39837)
0928 20:24:35.870221 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.870222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.870229 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:35.870277 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.870277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.870298 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.870369 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:35.870470 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.870965 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.871644 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.871923 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.871977 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.872125 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.872237 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.872293 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.872377 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.872394 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":122,"threads_started":1}
I20260928 20:24:35.878351 27675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.875449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48559 (local address 127.24.186.194:39501)
0928 20:24:35.875536 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.875538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.875581 (+ 43us) server_negotiation.cc:407] Connection header received
0928 20:24:35.875663 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.875671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.875683 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.875739 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:35.875911 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.876399 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.877396 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.877610 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.877701 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.878039 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.878096 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.878157 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.878223 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.878265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 20:24:35.878376 27691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.875336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39501 (local address 127.24.186.193:48559)
0928 20:24:35.875516 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:35.875530 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:35.875600 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:35.875742 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:35.875743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:35.875744 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:35.875852 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.875853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.876478 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.876479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.877274 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.877277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.877705 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.877707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.877778 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.877795 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.878159 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.878162 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:35.878211 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:24:35.878265 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260928 20:24:35.880657 27369 raft_consensus.cc:1261] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Refusing update from remote peer f23d6eb15f9f4a719b64116433aa8046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:35.880677 27509 raft_consensus.cc:1261] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Refusing update from remote peer f23d6eb15f9f4a719b64116433aa8046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:35.880924 27668 consensus_queue.cc:1035] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e96f3e9f564d3fb0fbd889070d7155" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:35.881052 27686 consensus_queue.cc:1035] T 41112f4e8c2f43299d84c676447c3ce6 P f23d6eb15f9f4a719b64116433aa8046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d605bf622b2643268c6d3c359b7c5aa8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:35.888840 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.886481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:34639 (local address 127.24.186.193:39837)
0928 20:24:35.886534 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:35.886536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:35.886660 (+ 124us) server_negotiation.cc:407] Connection header received
0928 20:24:35.886680 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:35.886680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:35.886688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:35.886722 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:35.886878 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.887360 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.888285 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.888518 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.888571 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.888679 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.888704 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.888733 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:35.888769 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:24:35.888783 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:24:35.888859 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:35.886416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39837 (local address 127.24.186.194:34639)
0928 20:24:35.886542 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:35.886552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:35.886607 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:35.886750 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:35.886751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:35.886753 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:35.886798 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.886800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.887426 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.887426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.888214 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:35.888217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:35.888579 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:35.888579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:35.888621 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:35.888625 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:35.888739 (+ 114us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:35.888740 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:24:35.888768 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 20:24:35.888800 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
W20260928 20:24:35.889190 27639 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 20:24:37.414355 27071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:37.414530 27071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:37.415475 27071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae5799c811b94c6cad1605ef8511c87a: Sending DeleteTablet for 3 replicas of tablet 41112f4e8c2f43299d84c676447c3ce6
I20260928 20:24:37.415751 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:37 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55140
I20260928 20:24:37.415764 27489 tablet_service.cc:1514] Processing DeleteTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:37 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:55410
I20260928 20:24:37.415823 27209 tablet_service.cc:1514] Processing DeleteTablet for tablet 41112f4e8c2f43299d84c676447c3ce6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:37 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57900
I20260928 20:24:37.415931 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27144
I20260928 20:24:37.417271 27720 tablet_replica.cc:330] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: stopping tablet replica
I20260928 20:24:37.417354 27720 raft_consensus.cc:2229] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:37.417466 27720 raft_consensus.cc:2258] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:37.417587 27721 tablet_replica.cc:330] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: stopping tablet replica
I20260928 20:24:37.417865 27720 ts_tablet_manager.cc:1905] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:37.419981 27720 ts_tablet_manager.cc:1918] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:24:37.420111 27720 log.cc:1196] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/41112f4e8c2f43299d84c676447c3ce6
I20260928 20:24:37.420385 27720 ts_tablet_manager.cc:1939] T 41112f4e8c2f43299d84c676447c3ce6 P d4e96f3e9f564d3fb0fbd889070d7155: Deleting consensus metadata
I20260928 20:24:37.420761 27721 raft_consensus.cc:2229] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:37.420753 27056 catalog_manager.cc:4841] TS d4e96f3e9f564d3fb0fbd889070d7155 (127.24.186.194:39501): tablet 41112f4e8c2f43299d84c676447c3ce6 (table test-table [id=54b48d1d67f7431abfaf1d569c07afdd]) successfully deleted
I20260928 20:24:37.420887 27721 raft_consensus.cc:2258] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:37.422260 27721 ts_tablet_manager.cc:1905] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 20:24:37.424829 27057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:39837 (error 108)
I20260928 20:24:37.424849 27721 ts_tablet_manager.cc:1918] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 20:24:37.424932 27721 log.cc:1196] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/41112f4e8c2f43299d84c676447c3ce6
I20260928 20:24:37.425011 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27284
I20260928 20:24:37.427686 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:37.425304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39837 (local address 127.0.0.1:57916)
0928 20:24:37.427589 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:37.427644 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2258,"thread_start_us":68,"threads_started":1}
W20260928 20:24:37.427846 27057 catalog_manager.cc:4584] TS f23d6eb15f9f4a719b64116433aa8046 (127.24.186.193:39837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41112f4e8c2f43299d84c676447c3ce6: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39837: connect: Connection refused (error 111)
I20260928 20:24:37.431162 27721 ts_tablet_manager.cc:1939] T 41112f4e8c2f43299d84c676447c3ce6 P d605bf622b2643268c6d3c359b7c5aa8: Deleting consensus metadata
I20260928 20:24:37.431553 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27424
I20260928 20:24:37.431663 27055 catalog_manager.cc:4841] TS d605bf622b2643268c6d3c359b7c5aa8 (127.24.186.195:44257): tablet 41112f4e8c2f43299d84c676447c3ce6 (table test-table [id=54b48d1d67f7431abfaf1d569c07afdd]) successfully deleted
I20260928 20:24:37.438678 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27039
2026-09-28T20:24:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5653 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:37 dist-test-slave-k477 krb5kdc[27727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:37 dist-test-slave-k477 krb5kdc[27727](info): set up 2 sockets
Sep 28 20:24:37 dist-test-slave-k477 krb5kdc[27727](info): commencing operation
krb5kdc: starting...
W20260928 20:24:38.504407 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:38 dist-test-slave-k477 krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:38.627731 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33295
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:38.716655 27740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:38.716886 27740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:38.716912 27740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:38.718160 27740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:38.718214 27740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:38.718228 27740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:38.718240 27740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:38.718252 27740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:38.719851 27740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:38.720124 27740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:38.720363 27740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:24:38.726075 27740 server_base.cc:1034] running on GCE node
W20260928 20:24:38.726109 27745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:38.726109 27746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:38.726257 27748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:38.726655 27740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:38.726975 27740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:38.728165 27740 hybrid_clock.cc:648] HybridClock initialized: now 1790627078728146 us; error 40 us; skew 500 ppm
Sep 28 20:24:38 dist-test-slave-k477 krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:38.730677 27740 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:38.731127 27740 webserver.cc:460] Webserver started at http://127.24.186.254:40681/ using document root <none> and password file <none>
I20260928 20:24:38.731355 27740 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:38.731421 27740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:38.731580 27740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:38.732752 27740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "4f6838a23df9461cb103bbf3bfdd4722"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "53ec6ce8e016d91090ba2c52b574e0a3"
server_key_iv: "e4b1f440058f33b86f2fde5f84586d1f"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.733121 27740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "4f6838a23df9461cb103bbf3bfdd4722"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "53ec6ce8e016d91090ba2c52b574e0a3"
server_key_iv: "e4b1f440058f33b86f2fde5f84586d1f"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.734464 27740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:38.735183 27755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:38.735405 27740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:38.735512 27740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "4f6838a23df9461cb103bbf3bfdd4722"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "53ec6ce8e016d91090ba2c52b574e0a3"
server_key_iv: "e4b1f440058f33b86f2fde5f84586d1f"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.735602 27740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:38.759699 27740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:38.762306 27740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:38.762486 27740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:38.766587 27740 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:33295
I20260928 20:24:38.766640 27807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:33295 every 8 connection(s)
I20260928 20:24:38.767036 27740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:38.768035 27808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:38.770416 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722: Bootstrap starting.
I20260928 20:24:38.771040 27808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:38.771349 27808 log.cc:824] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:38.772217 27808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722: No bootstrap required, opened a new log
I20260928 20:24:38.773748 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27740
I20260928 20:24:38.773876 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:38.773892 27808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } }
I20260928 20:24:38.774066 27808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:38.774060 25323 external_mini_cluster.cc:1428] Setting key 79c646c2ca3cf33aba9006789f5eca89
I20260928 20:24:38.774091 27808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f6838a23df9461cb103bbf3bfdd4722, State: Initialized, Role: FOLLOWER
I20260928 20:24:38.774282 27808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } }
I20260928 20:24:38.774376 27808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:38.774422 27808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:38.774477 27808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:38.775609 27808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } }
I20260928 20:24:38.775734 27808 leader_election.cc:302] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f6838a23df9461cb103bbf3bfdd4722; no voters:
I20260928 20:24:38.775930 27808 leader_election.cc:288] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:38.775983 27813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:38.776162 27813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [term 1 LEADER]: Becoming Leader. State: Replica: 4f6838a23df9461cb103bbf3bfdd4722, State: Running, Role: LEADER
I20260928 20:24:38.776221 27808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:38.776292 27813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } }
I20260928 20:24:38.776731 27813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f6838a23df9461cb103bbf3bfdd4722. Latest consensus state: current_term: 1 leader_uuid: "4f6838a23df9461cb103bbf3bfdd4722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } } }
I20260928 20:24:38.776737 27814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f6838a23df9461cb103bbf3bfdd4722" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f6838a23df9461cb103bbf3bfdd4722" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33295 } } }
I20260928 20:24:38.776898 27813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:38.776906 27814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:38.777277 27822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:38.778044 27822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:38.780999 27822 catalog_manager.cc:1347] Generated new cluster ID: e95ef75c44b54fefaf7640fa5f564f4f
I20260928 20:24:38.781061 27822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:24:38 dist-test-slave-k477 krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:38.785475 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:38.774646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.24.186.254:33295)
0928 20:24:38.774853 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:38.774864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:38.774901 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:24:38.775043 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:38.775051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:38.775118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:38.775308 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:38.775963 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.777484 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.778441 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.779050 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.779177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:38.782010 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:38.782014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:38.782014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:38.782031 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:38.783743 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:38.784175 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:38.784178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:38.784178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:38.784286 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:38.784489 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:38.784490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:38.784490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:38.784756 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:38.784947 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:38.785021 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 20:24:38.785185 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260928 20:24:38.799919 27822 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:38.800536 27822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:38.806836 27822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f6838a23df9461cb103bbf3bfdd4722: Generated new TSK 0
I20260928 20:24:38.807120 27822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:38.843438 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:38.933037 27834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:38.933274 27834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:38.933300 27834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:38.934566 27834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:38.934623 27834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:38.934653 27834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:38.936432 27834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:38.936728 27834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:38.936995 27834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:24:38.942531 27834 server_base.cc:1034] running on GCE node
W20260928 20:24:38.942456 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:38.942482 27842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:38.942463 27839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:38.942924 27834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:38.943205 27834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:38.944378 27834 hybrid_clock.cc:648] HybridClock initialized: now 1790627078944364 us; error 36 us; skew 500 ppm
Sep 28 20:24:38 dist-test-slave-k477 krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:38.947983 27834 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:38.948555 27834 webserver.cc:460] Webserver started at http://127.24.186.193:41781/ using document root <none> and password file <none>
I20260928 20:24:38.948810 27834 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:38.948877 27834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:38.949023 27834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:38.950126 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "d91ce01b4bba4a7fa7b485155f2a5866"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "e742e197a197441d8ced9a42bec8de17"
server_key_iv: "ad45d568172d163436ef6485700a7348"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.950498 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "d91ce01b4bba4a7fa7b485155f2a5866"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "e742e197a197441d8ced9a42bec8de17"
server_key_iv: "ad45d568172d163436ef6485700a7348"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.951987 27834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:38.952764 27849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:38.952976 27834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:38.953082 27834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "d91ce01b4bba4a7fa7b485155f2a5866"
format_stamp: "Formatted at 2026-09-28 20:24:38 on dist-test-slave-k477"
server_key: "e742e197a197441d8ced9a42bec8de17"
server_key_iv: "ad45d568172d163436ef6485700a7348"
server_key_version: "encryptionkey@0"
I20260928 20:24:38.953164 27834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:38.967576 27834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:38.970923 27834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:38.971166 27834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:38.971490 27834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:38.972497 27834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:38.972607 27834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:38.972654 27834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:38.972849 27834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:38.980530 27834 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35775
I20260928 20:24:38.980577 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35775 every 8 connection(s)
I20260928 20:24:38.980988 27834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:38.989403 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27834
I20260928 20:24:38.989562 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:38.989876 25323 external_mini_cluster.cc:1428] Setting key cd68cbbd8bbd6e37a6c7b06894e2f43d
Sep 28 20:24:38 dist-test-slave-k477 krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627078, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:38.994635 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:38.981886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:49345 (local address 127.24.186.254:33295)
0928 20:24:38.982062 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:38.982065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:38.982181 (+ 116us) server_negotiation.cc:407] Connection header received
0928 20:24:38.982721 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:38.982722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:38.982739 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:38.982809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:38.983912 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.984469 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.986050 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.986277 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.986351 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:38.990709 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:38.990712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:38.990712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:38.990722 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:38.992485 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:38.993035 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:38.993037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:38.993037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:38.993105 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:38.993406 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:38.993407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:38.993407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:38.993546 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:38.993618 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:38.994403 (+ 785us) server_negotiation.cc:299] Negotiation successful
0928 20:24:38.994516 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260928 20:24:38.994944 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:38.981834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33295 (local address 127.24.186.193:49345)
0928 20:24:38.982016 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:38.982065 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:38.982614 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:38.982874 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:38.982878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:38.983528 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:38.983752 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:38.983756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.984653 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.984655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:38.985928 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:38.985935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:38.986497 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:38.986499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:38.986689 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:38.987565 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:38.987588 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:38.990554 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:38.992622 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:38.992636 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:38.992640 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:38.992946 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:38.993192 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:38.993194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:38.993194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:38.993330 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:38.993644 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:38.993651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:38.994097 (+ 446us) client_negotiation.cc:769] Sending connection context
0928 20:24:38.994379 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 20:24:38.994626 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 20:24:38.995553 27963 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33295
I20260928 20:24:38.995666 27963 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:38.995927 27963 heartbeater.cc:511] Master 127.24.186.254:33295 requested a full tablet report, sending...
I20260928 20:24:38.996618 27772 ts_manager.cc:194] Registered new tserver with Master: d91ce01b4bba4a7fa7b485155f2a5866 (127.24.186.193:35775)
I20260928 20:24:38.997447 27772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:49345
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:39.030078 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:39.122694 27968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:39.122958 27968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:39.122997 27968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:39.124239 27968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:39.124311 27968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:39.124358 27968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:39.125855 27968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:39.126143 27968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:39.126394 27968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:39.131410 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:39.131417 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:39.131417 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:39.131690 27968 server_base.cc:1034] running on GCE node
I20260928 20:24:39.131958 27968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:39.132210 27968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:39.133368 27968 hybrid_clock.cc:648] HybridClock initialized: now 1790627079133356 us; error 28 us; skew 500 ppm
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627079, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:39.136282 27968 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:39.136900 27968 webserver.cc:460] Webserver started at http://127.24.186.194:44607/ using document root <none> and password file <none>
I20260928 20:24:39.137180 27968 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:39.137252 27968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:39.137396 27968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:39.138900 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "b967d8db3a1c432689151b94af10830a"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "70ce315bd3aae4582867e365955dba38"
server_key_iv: "d61e854055f72c29966da039fd2d1908"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.139297 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "b967d8db3a1c432689151b94af10830a"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "70ce315bd3aae4582867e365955dba38"
server_key_iv: "d61e854055f72c29966da039fd2d1908"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.140820 27968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260928 20:24:39.141635 27983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.141878 27968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:39.141979 27968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "b967d8db3a1c432689151b94af10830a"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "70ce315bd3aae4582867e365955dba38"
server_key_iv: "d61e854055f72c29966da039fd2d1908"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.142063 27968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:39.159996 27968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:39.162359 27968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:39.162532 27968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:39.162783 27968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:39.163095 27968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:39.163146 27968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.163187 27968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:39.163225 27968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.170112 27968 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:38265
I20260928 20:24:39.170186 28096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:38265 every 8 connection(s)
I20260928 20:24:39.170547 27968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:39.177141 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 27968
I20260928 20:24:39.177273 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:39.177470 25323 external_mini_cluster.cc:1428] Setting key 5ae41b71f980ce72024dc94fbf779012
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627079, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:39.182633 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:39.171517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33295 (local address 127.24.186.194:40067)
0928 20:24:39.171676 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:39.171721 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:39.172293 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:39.172603 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:39.172606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:39.173238 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:39.173463 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:39.173468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.174428 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.174429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:39.175312 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:39.175314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.176051 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.176052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:39.176213 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:39.176847 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:39.176856 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:39.178549 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:39.180719 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:39.180727 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:39.180730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:39.181094 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:39.181337 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:39.181338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:39.181338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:39.181475 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:39.181815 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:39.181821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:39.182006 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 20:24:39.182117 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:24:39.182313 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260928 20:24:39.183080 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:39.171538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40067 (local address 127.24.186.254:33295)
0928 20:24:39.171606 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:39.171608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:39.171939 (+ 331us) server_negotiation.cc:407] Connection header received
0928 20:24:39.172421 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:39.172422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:39.172432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:39.172501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:39.173618 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.174250 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.175520 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.175955 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.176029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:39.178658 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:39.178660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:39.178660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:39.178667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:39.180593 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:39.181194 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:39.181195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:39.181196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:39.181261 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:39.181549 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:39.181550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:39.181551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:39.181739 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:39.182805 (+ 1066us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:39.182857 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 20:24:39.182967 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:24:39.185118 28097 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33295
I20260928 20:24:39.185243 28097 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:39.185470 28097 heartbeater.cc:511] Master 127.24.186.254:33295 requested a full tablet report, sending...
I20260928 20:24:39.186396 27772 ts_manager.cc:194] Registered new tserver with Master: b967d8db3a1c432689151b94af10830a (127.24.186.194:38265)
I20260928 20:24:39.186918 27772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:40067
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:39.207552 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:39.290580 28102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:39.290858 28102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:39.290899 28102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:39.292196 28102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:39.292259 28102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:39.292310 28102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:39.294034 28102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:39.294420 28102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:39.294689 28102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:39.299708 28107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:39.299865 28102 server_base.cc:1034] running on GCE node
W20260928 20:24:39.299731 28108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:39.299968 28110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:39.300216 28102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:39.300427 28102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:39.301591 28102 hybrid_clock.cc:648] HybridClock initialized: now 1790627079301577 us; error 29 us; skew 500 ppm
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627079, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:39.304379 28102 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:39.304840 28102 webserver.cc:460] Webserver started at http://127.24.186.195:44183/ using document root <none> and password file <none>
I20260928 20:24:39.305096 28102 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:39.305164 28102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:39.305315 28102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:39.306416 28102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "ff5b0ca6ff9b4ee9a75f5bdb620029ef"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "7b03bfe9c25ea8aa203a2308469b5c62"
server_key_iv: "466dc7188eede82198476511c731e532"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.306919 28102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "ff5b0ca6ff9b4ee9a75f5bdb620029ef"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "7b03bfe9c25ea8aa203a2308469b5c62"
server_key_iv: "466dc7188eede82198476511c731e532"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.308773 28102 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:24:39.309691 28117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.309937 28102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:39.310063 28102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "ff5b0ca6ff9b4ee9a75f5bdb620029ef"
format_stamp: "Formatted at 2026-09-28 20:24:39 on dist-test-slave-k477"
server_key: "7b03bfe9c25ea8aa203a2308469b5c62"
server_key_iv: "466dc7188eede82198476511c731e532"
server_key_version: "encryptionkey@0"
I20260928 20:24:39.310178 28102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:39.331473 28102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:39.334309 28102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:39.334465 28102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:39.334702 28102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:39.335088 28102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:39.335171 28102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.335220 28102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:39.335261 28102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:39.341780 28102 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38555
I20260928 20:24:39.341842 28230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38555 every 8 connection(s)
I20260928 20:24:39.342195 28102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:39.343091 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28102
I20260928 20:24:39.343179 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:39.343381 25323 external_mini_cluster.cc:1428] Setting key 512995c3e87482800a1009226cb17648
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627079, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:39.352825 27812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:39.343025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:54593 (local address 127.24.186.254:33295)
0928 20:24:39.343097 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:39.343103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:39.343399 (+ 296us) server_negotiation.cc:407] Connection header received
0928 20:24:39.344041 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:39.344043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:39.344055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:39.344118 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:39.345189 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.345690 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.346750 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.346951 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.347037 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:39.349338 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:39.349340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:39.349340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:39.349346 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:39.350631 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:39.351486 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:39.351489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:39.351489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:39.351622 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:39.352100 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:39.352102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:39.352103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:39.352288 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:39.352362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:39.352669 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 20:24:39.352750 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:39.352918 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:39.342993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33295 (local address 127.24.186.195:54593)
0928 20:24:39.343264 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:39.343310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:39.343938 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:39.344196 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:39.344199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:39.344840 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:39.345100 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:39.345105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.345790 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.345791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:39.346660 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:39.346663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:39.347057 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:39.347058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:39.347178 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:39.347672 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:39.347680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:39.349248 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:39.350729 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:39.350737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:39.350739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:39.350985 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:39.351758 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:39.351759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:39.351760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:39.351984 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:39.352362 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:39.352367 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:39.352502 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 20:24:39.352589 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:24:39.352737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260928 20:24:39.353523 28231 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33295
I20260928 20:24:39.353641 28231 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:39.353835 28231 heartbeater.cc:511] Master 127.24.186.254:33295 requested a full tablet report, sending...
I20260928 20:24:39.354269 27772 ts_manager.cc:194] Registered new tserver with Master: ff5b0ca6ff9b4ee9a75f5bdb620029ef (127.24.186.195:38555)
I20260928 20:24:39.354743 27772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:54593
I20260928 20:24:39.355109 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 20:24:39.368960 27812 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:39.367630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49416 (local address 127.24.186.254:33295)
0928 20:24:39.367695 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:39.367697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:39.367737 (+ 40us) server_negotiation.cc:407] Connection header received
0928 20:24:39.368033 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:39.368035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:39.368058 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:39.368161 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:39.368869 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49416: BlockingRecv error: recv got EOF from 127.0.0.1:49416 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:24:39.369779 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27834
I20260928 20:24:39.374833 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27968
I20260928 20:24:39.379906 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28102
I20260928 20:24:39.384090 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 27740
2026-09-28T20:24:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1942 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[28243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[28243](info): set up 2 sockets
Sep 28 20:24:39 dist-test-slave-k477 krb5kdc[28243](info): commencing operation
krb5kdc: starting...
W20260928 20:24:40.449250 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:40.576010 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46837
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:46837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:40.659365 28256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:40.659627 28256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:40.659667 28256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:40.660863 28256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:40.660931 28256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:40.660956 28256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:40.660984 28256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:40.661013 28256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:40.662696 28256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:46837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:40.663012 28256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:40.663270 28256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:40.668249 28261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:40.668292 28264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:40.668292 28262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:40.668392 28256 server_base.cc:1034] running on GCE node
I20260928 20:24:40.668684 28256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:40.669039 28256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:40.670207 28256 hybrid_clock.cc:648] HybridClock initialized: now 1790627080670199 us; error 40 us; skew 500 ppm
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627080, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:40.672848 28256 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:40.673394 28256 webserver.cc:460] Webserver started at http://127.24.186.254:36365/ using document root <none> and password file <none>
I20260928 20:24:40.673648 28256 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:40.673717 28256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:40.673868 28256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:40.674980 28256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "6876202b1c3a41e89cb572a95171787f"
format_stamp: "Formatted at 2026-09-28 20:24:40 on dist-test-slave-k477"
server_key: "869ca8ac4c62679f994f5bdbdef93aba"
server_key_iv: "e8084eb072944834e9a9a8747525fc0d"
server_key_version: "encryptionkey@0"
I20260928 20:24:40.675350 28256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "6876202b1c3a41e89cb572a95171787f"
format_stamp: "Formatted at 2026-09-28 20:24:40 on dist-test-slave-k477"
server_key: "869ca8ac4c62679f994f5bdbdef93aba"
server_key_iv: "e8084eb072944834e9a9a8747525fc0d"
server_key_version: "encryptionkey@0"
I20260928 20:24:40.677179 28256 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 20:24:40.678489 28271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:40.679014 28256 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:24:40.679201 28256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "6876202b1c3a41e89cb572a95171787f"
format_stamp: "Formatted at 2026-09-28 20:24:40 on dist-test-slave-k477"
server_key: "869ca8ac4c62679f994f5bdbdef93aba"
server_key_iv: "e8084eb072944834e9a9a8747525fc0d"
server_key_version: "encryptionkey@0"
I20260928 20:24:40.679315 28256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:40.691020 28256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:40.693373 28256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:40.693537 28256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:40.698398 28256 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:46837
I20260928 20:24:40.698438 28323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:46837 every 8 connection(s)
I20260928 20:24:40.698943 28256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:40.699609 28324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:40.701333 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28256
I20260928 20:24:40.701436 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:40.701601 25323 external_mini_cluster.cc:1428] Setting key acb6828666484db5b36571f1f4d31090
I20260928 20:24:40.702333 28324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f: Bootstrap starting.
I20260928 20:24:40.702960 28324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:40.703325 28324 log.cc:824] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:40.704103 28324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f: No bootstrap required, opened a new log
I20260928 20:24:40.706313 28324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } }
I20260928 20:24:40.706460 28324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:40.706482 28324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6876202b1c3a41e89cb572a95171787f, State: Initialized, Role: FOLLOWER
I20260928 20:24:40.706670 28324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } }
I20260928 20:24:40.706755 28324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:40.706804 28324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:40.706864 28324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:40.707651 28324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } }
I20260928 20:24:40.707778 28324 leader_election.cc:302] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6876202b1c3a41e89cb572a95171787f; no voters:
I20260928 20:24:40.707983 28324 leader_election.cc:288] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:40.708050 28329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:40.708257 28329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [term 1 LEADER]: Becoming Leader. State: Replica: 6876202b1c3a41e89cb572a95171787f, State: Running, Role: LEADER
I20260928 20:24:40.708283 28324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:40.708405 28329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } }
I20260928 20:24:40.708814 28330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6876202b1c3a41e89cb572a95171787f. Latest consensus state: current_term: 1 leader_uuid: "6876202b1c3a41e89cb572a95171787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } } }
I20260928 20:24:40.708900 28330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [sys.catalog]: This master's current role is: LEADER
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:40.709448 28335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:40.709968 28335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:40.710088 28331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6876202b1c3a41e89cb572a95171787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6876202b1c3a41e89cb572a95171787f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46837 } } }
I20260928 20:24:40.710242 28331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:40.711499 28335 catalog_manager.cc:1347] Generated new cluster ID: d9f06895e80c4a0caf90d10d573206dc
I20260928 20:24:40.711550 28335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:40.713654 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:40.702139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.24.186.254:46837)
0928 20:24:40.702315 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:40.702322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:40.702348 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:24:40.702788 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:40.702795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:40.702842 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:40.702960 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:40.703775 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:40.704805 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:40.705996 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:40.706424 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:40.706578 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:40.709582 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:40.709587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:40.709588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:40.709603 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:40.711928 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:40.712338 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:40.712341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:40.712342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:40.712439 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:40.712681 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:40.712683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:40.712683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:40.712941 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:40.713122 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:40.713227 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:24:40.713484 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 20:24:40.714628 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:40.726392 28335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:40.727310 28335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:40.729427 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:40.725949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.24.186.254:46837)
0928 20:24:40.725991 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:40.725993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:40.726051 (+ 58us) server_negotiation.cc:407] Connection header received
0928 20:24:40.726198 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:40.726200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:40.726214 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:40.726296 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:40.726707 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:40.727200 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:40.728030 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:40.728227 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:40.728296 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:40.728975 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:40.728977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:40.728977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:40.729164 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:40.729180 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:40.729186 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:40.729245 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:40.729314 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 20:24:40.729352 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:24:40.739998 28335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6876202b1c3a41e89cb572a95171787f: Generated new TSK 0
I20260928 20:24:40.740268 28335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:40.749114 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28256
2026-09-28T20:24:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1363 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28357](info): set up 2 sockets
Sep 28 20:24:40 dist-test-slave-k477 krb5kdc[28357](info): commencing operation
krb5kdc: starting...
W20260928 20:24:41.812561 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:41 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:41.940356 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44429
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:44429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:42.025158 28370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:42.025391 28370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:42.025416 28370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:42.026685 28370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:42.026736 28370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:42.026751 28370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:42.026765 28370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:42.026777 28370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:42.028391 28370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:44429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:42.028661 28370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:42.028896 28370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:42.034018 28375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:42.034013 28376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.034173 28370 server_base.cc:1034] running on GCE node
W20260928 20:24:42.034013 28378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.034538 28370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:42.034838 28370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:42.036017 28370 hybrid_clock.cc:648] HybridClock initialized: now 1790627082035998 us; error 36 us; skew 500 ppm
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:42.038727 28370 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:42.039148 28370 webserver.cc:460] Webserver started at http://127.24.186.254:44315/ using document root <none> and password file <none>
I20260928 20:24:42.039389 28370 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:42.039436 28370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:42.039594 28370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:42.040794 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "887d1cd1ee6946c4afd619f65b7e34e1"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "89200ca294bb28b4db427694d2635117"
server_key_iv: "cb70351805baf1cc7cc317e326a3d14d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.041178 28370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "887d1cd1ee6946c4afd619f65b7e34e1"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "89200ca294bb28b4db427694d2635117"
server_key_iv: "cb70351805baf1cc7cc317e326a3d14d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.042652 28370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:42.043574 28385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.043793 28370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:42.043918 28370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "887d1cd1ee6946c4afd619f65b7e34e1"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "89200ca294bb28b4db427694d2635117"
server_key_iv: "cb70351805baf1cc7cc317e326a3d14d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.043988 28370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:42.056936 28370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:42.059103 28370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:42.059262 28370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:42.064479 28370 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:44429
I20260928 20:24:42.064512 28437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:44429 every 8 connection(s)
I20260928 20:24:42.064868 28370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:42.065917 28438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:42.067363 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28370
I20260928 20:24:42.067472 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:42.067654 25323 external_mini_cluster.cc:1428] Setting key a30a2688be91029ef1685cbef8497b3d
I20260928 20:24:42.068812 28438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1: Bootstrap starting.
I20260928 20:24:42.069510 28438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:42.069808 28438 log.cc:824] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:42.070621 28438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1: No bootstrap required, opened a new log
I20260928 20:24:42.073247 28438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } }
I20260928 20:24:42.073436 28438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:42.073464 28438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887d1cd1ee6946c4afd619f65b7e34e1, State: Initialized, Role: FOLLOWER
I20260928 20:24:42.073613 28438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } }
I20260928 20:24:42.073719 28438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:42.073766 28438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:42.073819 28438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:42.074446 28438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } }
I20260928 20:24:42.074533 28438 leader_election.cc:302] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 887d1cd1ee6946c4afd619f65b7e34e1; no voters:
I20260928 20:24:42.074784 28438 leader_election.cc:288] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:42.074806 28443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:42.074935 28443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [term 1 LEADER]: Becoming Leader. State: Replica: 887d1cd1ee6946c4afd619f65b7e34e1, State: Running, Role: LEADER
I20260928 20:24:42.075018 28443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } }
I20260928 20:24:42.075088 28438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:42.075338 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } } }
I20260928 20:24:42.075345 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 887d1cd1ee6946c4afd619f65b7e34e1. Latest consensus state: current_term: 1 leader_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "887d1cd1ee6946c4afd619f65b7e34e1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44429 } } }
I20260928 20:24:42.075426 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:42.075440 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1 [sys.catalog]: This master's current role is: LEADER
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:42.075809 28447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:42.076406 28447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:42.077965 28447 catalog_manager.cc:1347] Generated new cluster ID: b51d417feb1d4897957f98be1903686d
I20260928 20:24:42.078018 28447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:42.080260 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.068229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.24.186.254:44429)
0928 20:24:42.068391 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:42.068401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:42.068432 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:24:42.068943 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:42.068952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:42.069010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:42.069189 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:42.070128 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.071492 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.072403 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.073123 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.073257 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.076163 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:42.076168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:42.076169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:42.076184 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:42.078447 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.078896 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.078899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.078899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.079014 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.079213 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.079214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.079214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.079526 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:42.079678 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:42.079797 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 20:24:42.079983 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260928 20:24:42.084697 28447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:42.085224 28447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:42.090561 28447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 887d1cd1ee6946c4afd619f65b7e34e1: Generated new TSK 0
I20260928 20:24:42.090787 28447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:42.119609 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:42.206491 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:42.206717 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:42.206743 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:42.208014 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:42.208063 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:42.208092 28464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:42.209623 28464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:42.209878 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:42.210125 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:42.215231 28470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:42.215220 28472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.215523 28464 server_base.cc:1034] running on GCE node
W20260928 20:24:42.215220 28469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.215956 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:42.216245 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:42.217481 28464 hybrid_clock.cc:648] HybridClock initialized: now 1790627082217448 us; error 28 us; skew 500 ppm
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:42.220299 28464 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:42.220716 28464 webserver.cc:460] Webserver started at http://127.24.186.193:38073/ using document root <none> and password file <none>
I20260928 20:24:42.220916 28464 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:42.220956 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:42.221047 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:42.222153 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "86eaaff06df14ed7a08ad42378be5320"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "ff7911f6187811b086803630a5c84590"
server_key_iv: "d9d877cca7290db0db6515566a4ba28c"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.222496 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "86eaaff06df14ed7a08ad42378be5320"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "ff7911f6187811b086803630a5c84590"
server_key_iv: "d9d877cca7290db0db6515566a4ba28c"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.223924 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:42.224694 28479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.224876 28464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:42.224949 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "86eaaff06df14ed7a08ad42378be5320"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "ff7911f6187811b086803630a5c84590"
server_key_iv: "d9d877cca7290db0db6515566a4ba28c"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.225005 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:42.236770 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:42.239084 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:42.239208 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:42.239398 28464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:42.239713 28464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:42.239748 28464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.239773 28464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:42.239784 28464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.246080 28464 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:44691
I20260928 20:24:42.246126 28592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:44691 every 8 connection(s)
I20260928 20:24:42.246455 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:42.255084 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28464
I20260928 20:24:42.255232 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:42.255466 25323 external_mini_cluster.cc:1428] Setting key d5533bdc32523b9aacaa1c1a8fe26fba
I20260928 20:24:42.256847 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.247154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:42951 (local address 127.24.186.254:44429)
0928 20:24:42.247195 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:42.247197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:42.247409 (+ 212us) server_negotiation.cc:407] Connection header received
0928 20:24:42.248725 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:42.248727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:42.248740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:42.248790 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:42.249495 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.250027 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.251036 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.251274 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.251360 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.253590 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:42.253592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:42.253592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:42.253600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:42.255255 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.255825 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.255827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.255827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.255899 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.256172 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.256173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.256174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.256297 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:42.256377 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:42.256656 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 20:24:42.256755 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:24:42.257057 28595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.247155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44429 (local address 127.24.186.193:42951)
0928 20:24:42.247290 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:42.247326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:42.248545 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:42.248866 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:42.248870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:42.249131 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:42.249347 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.249355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.250124 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.250125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.250932 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.250934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.251372 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.251373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.251491 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.252114 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:42.252123 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:42.253482 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:42.255349 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.255359 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.255362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.255708 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.255974 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.255975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.255976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.256099 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.256367 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:42.256373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:42.256523 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 20:24:42.256639 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:24:42.256803 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260928 20:24:42.257611 28593 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44429
I20260928 20:24:42.257732 28593 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:42.257916 28593 heartbeater.cc:511] Master 127.24.186.254:44429 requested a full tablet report, sending...
I20260928 20:24:42.258483 28402 ts_manager.cc:194] Registered new tserver with Master: 86eaaff06df14ed7a08ad42378be5320 (127.24.186.193:44691)
I20260928 20:24:42.259123 28402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:42951
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:42.287587 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:42.380226 28598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:42.380511 28598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:42.380561 28598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:42.382025 28598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:42.382109 28598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:42.383136 28598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:42.385064 28598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:42.385392 28598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:42.385665 28598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:42.390866 28604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:42.390887 28606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:42.390944 28603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.391248 28598 server_base.cc:1034] running on GCE node
I20260928 20:24:42.391464 28598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:42.391682 28598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:42.392843 28598 hybrid_clock.cc:648] HybridClock initialized: now 1790627082392826 us; error 30 us; skew 500 ppm
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:42.396458 28598 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:42.397248 28598 webserver.cc:460] Webserver started at http://127.24.186.194:43617/ using document root <none> and password file <none>
I20260928 20:24:42.397680 28598 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:42.397749 28598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:42.397917 28598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:42.399041 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "2401a100541642cb8f007409f8c2db09"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "aa9201e53de17b4e5537571842e5cb86"
server_key_iv: "b59f1345860bac19c713b276aa1d2c6e"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.399420 28598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "2401a100541642cb8f007409f8c2db09"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "aa9201e53de17b4e5537571842e5cb86"
server_key_iv: "b59f1345860bac19c713b276aa1d2c6e"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.401016 28598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:42.401958 28613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.402961 28598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:24:42.403057 28598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "2401a100541642cb8f007409f8c2db09"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "aa9201e53de17b4e5537571842e5cb86"
server_key_iv: "b59f1345860bac19c713b276aa1d2c6e"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.403177 28598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:42.425244 28598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:42.427614 28598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:42.427773 28598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:42.428093 28598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:42.428364 28598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:42.428393 28598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.428416 28598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:42.428428 28598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.434818 28598 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:42737
I20260928 20:24:42.435040 28726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:42737 every 8 connection(s)
I20260928 20:24:42.435248 28598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:42.444669 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28598
I20260928 20:24:42.444797 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:42.444867 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.436077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:51553 (local address 127.24.186.254:44429)
0928 20:24:42.436165 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:42.436167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:42.436365 (+ 198us) server_negotiation.cc:407] Connection header received
0928 20:24:42.436913 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:42.436915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:42.436926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:42.436989 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:42.437931 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.438440 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.439443 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.439630 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.439705 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.442169 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:42.442172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:42.442172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:42.442179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:42.443424 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.443891 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.443896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.443897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.443956 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.444193 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.444194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.444194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.444311 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:42.444366 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:42.444659 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 20:24:42.444751 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 20:24:42.445010 25323 external_mini_cluster.cc:1428] Setting key 80b82bcf17cb51647f1d7d3268cfe1ac
I20260928 20:24:42.445005 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.436044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44429 (local address 127.24.186.194:51553)
0928 20:24:42.436211 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:42.436269 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:42.436793 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:42.437042 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:42.437045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:42.437633 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:42.437828 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.437833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.438525 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.438526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.439350 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.439352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.439737 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.439738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.439913 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.440393 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:42.440402 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:42.442036 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:42.443506 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.443512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.443514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.443751 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.444025 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.444025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.444026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.444133 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.444370 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:42.444374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:42.444513 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 20:24:42.444595 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 20:24:42.444720 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 20:24:42.445629 28727 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44429
I20260928 20:24:42.445739 28727 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:42.445935 28727 heartbeater.cc:511] Master 127.24.186.254:44429 requested a full tablet report, sending...
I20260928 20:24:42.446339 28402 ts_manager.cc:194] Registered new tserver with Master: 2401a100541642cb8f007409f8c2db09 (127.24.186.194:42737)
I20260928 20:24:42.446795 28402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:51553
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:42.472273 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:42.559911 28732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:42.560125 28732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:42.560149 28732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:42.561477 28732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:42.561528 28732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:42.561560 28732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:42.563011 28732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:42.563207 28732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:42.563519 28732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:42.569190 28737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.569204 28732 server_base.cc:1034] running on GCE node
W20260928 20:24:42.569187 28738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:42.569187 28740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:42.569674 28732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:42.569883 28732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:42.571072 28732 hybrid_clock.cc:648] HybridClock initialized: now 1790627082571057 us; error 29 us; skew 500 ppm
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:42.578962 28732 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:42.579396 28732 webserver.cc:460] Webserver started at http://127.24.186.195:45867/ using document root <none> and password file <none>
I20260928 20:24:42.579581 28732 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:42.579619 28732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:42.579713 28732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:42.580936 28732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "e7243bf42f7341d68a4c6edbc2fef26d"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "e7007ad0fef3acb60f211c876c6dc005"
server_key_iv: "fa3761cba40e0ea165c619b8a1f9bb7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.581243 28732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "e7243bf42f7341d68a4c6edbc2fef26d"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "e7007ad0fef3acb60f211c876c6dc005"
server_key_iv: "fa3761cba40e0ea165c619b8a1f9bb7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.582968 28732 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:24:42.583865 28747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.584029 28732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:42.584143 28732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "e7243bf42f7341d68a4c6edbc2fef26d"
format_stamp: "Formatted at 2026-09-28 20:24:42 on dist-test-slave-k477"
server_key: "e7007ad0fef3acb60f211c876c6dc005"
server_key_iv: "fa3761cba40e0ea165c619b8a1f9bb7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:42.584242 28732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:42.599941 28732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:42.602049 28732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:42.602249 28732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:42.602504 28732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:42.603596 28732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:42.603688 28732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.603739 28732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:42.603762 28732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:42.610309 28732 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38329
I20260928 20:24:42.610483 28860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38329 every 8 connection(s)
I20260928 20:24:42.610747 28732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:42.618111 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28732
I20260928 20:24:42.618232 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:42.618444 25323 external_mini_cluster.cc:1428] Setting key cd2a50fad4d9869c250b36ad4647ea2f
I20260928 20:24:42.621402 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.611572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:38769 (local address 127.24.186.254:44429)
0928 20:24:42.611639 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:42.611642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:42.611968 (+ 326us) server_negotiation.cc:407] Connection header received
0928 20:24:42.612521 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:42.612522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:42.612531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:42.612584 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:42.613535 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.614061 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.615060 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.615264 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.615320 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.617512 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:42.617514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:42.617514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:42.617521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:42.619081 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.620086 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.620089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.620089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.620196 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.620603 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.620605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.620605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.620769 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:42.620873 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:42.621217 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 20:24:42.621308 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:24:42.621640 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.611533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44429 (local address 127.24.186.195:38769)
0928 20:24:42.611741 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:42.611778 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:42.612433 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:42.612636 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:42.612639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:42.613235 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:42.613432 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.613437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.614153 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.614154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.614978 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:42.614981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.615344 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.615345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:42.615459 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.616055 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:42.616063 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:42.617422 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:42.619144 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.619151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.619153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.619919 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.620285 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:42.620286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:42.620287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:42.620426 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:42.620884 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:42.620893 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:42.621065 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 20:24:42.621180 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 20:24:42.621369 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
I20260928 20:24:42.622141 28861 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44429
I20260928 20:24:42.622244 28861 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:42.622408 28861 heartbeater.cc:511] Master 127.24.186.254:44429 requested a full tablet report, sending...
I20260928 20:24:42.622803 28402 ts_manager.cc:194] Registered new tserver with Master: e7243bf42f7341d68a4c6edbc2fef26d (127.24.186.195:38329)
I20260928 20:24:42.623148 28402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:38769
I20260928 20:24:42.630388 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627082, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:42.657193 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:42.648825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.24.186.254:44429)
0928 20:24:42.648892 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:42.648894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:42.648999 (+ 105us) server_negotiation.cc:407] Connection header received
0928 20:24:42.649063 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:42.649064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:42.649073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:42.649119 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:42.649763 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.650284 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.651049 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:42.651258 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:42.651332 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:42.654817 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:42.654827 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:42.654827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:42.654835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:42.656151 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.656496 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.656496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.656497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.656544 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:42.656709 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:42.656709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:42.656710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:42.656862 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:42.656940 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:42.657059 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 20:24:42.657125 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 20:24:42.657424 28402 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53994
I20260928 20:24:42.657927 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28464
I20260928 20:24:42.663368 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28598
I20260928 20:24:42.668329 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28732
I20260928 20:24:42.673003 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28370
2026-09-28T20:24:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1925 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28873](info): set up 2 sockets
Sep 28 20:24:42 dist-test-slave-k477 krb5kdc[28873](info): commencing operation
krb5kdc: starting...
W20260928 20:24:43.728583 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:43 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:43.858750 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41831
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:43.947320 28886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:43.947562 28886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:43.947587 28886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:43.948848 28886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:43.948904 28886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:43.948917 28886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:43.948930 28886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:43.948943 28886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:43.950577 28886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:43.950865 28886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:43.951116 28886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:43.956604 28891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:43.956640 28894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:43.956604 28892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:43.956996 28886 server_base.cc:1034] running on GCE node
I20260928 20:24:43.957304 28886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:43.957616 28886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:43.958799 28886 hybrid_clock.cc:648] HybridClock initialized: now 1790627083958778 us; error 46 us; skew 500 ppm
Sep 28 20:24:43 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627083, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:43.961490 28886 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:43.961994 28886 webserver.cc:460] Webserver started at http://127.24.186.254:33965/ using document root <none> and password file <none>
I20260928 20:24:43.962200 28886 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:43.962244 28886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:43.962349 28886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:43.963575 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "8422b1c5a61c48a0bcc88621c6773827"
format_stamp: "Formatted at 2026-09-28 20:24:43 on dist-test-slave-k477"
server_key: "521f3b2a4b8551ed24488c7ef3fe6a24"
server_key_iv: "721c5395f881eac057267925093b5017"
server_key_version: "encryptionkey@0"
I20260928 20:24:43.964277 28886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "8422b1c5a61c48a0bcc88621c6773827"
format_stamp: "Formatted at 2026-09-28 20:24:43 on dist-test-slave-k477"
server_key: "521f3b2a4b8551ed24488c7ef3fe6a24"
server_key_iv: "721c5395f881eac057267925093b5017"
server_key_version: "encryptionkey@0"
I20260928 20:24:43.966228 28886 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 20:24:43.967047 28901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:43.967263 28886 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:43.967382 28886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "8422b1c5a61c48a0bcc88621c6773827"
format_stamp: "Formatted at 2026-09-28 20:24:43 on dist-test-slave-k477"
server_key: "521f3b2a4b8551ed24488c7ef3fe6a24"
server_key_iv: "721c5395f881eac057267925093b5017"
server_key_version: "encryptionkey@0"
I20260928 20:24:43.967469 28886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:43.979487 28886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:43.981894 28886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:43.982056 28886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:43.986644 28886 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41831
I20260928 20:24:43.986677 28953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41831 every 8 connection(s)
I20260928 20:24:43.987319 28886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:43.988442 28954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:43.991328 28954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827: Bootstrap starting.
I20260928 20:24:43.992101 28954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:43.992396 28954 log.cc:824] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:43.993546 28954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827: No bootstrap required, opened a new log
I20260928 20:24:43.995025 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28886
I20260928 20:24:43.995174 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:43.995383 25323 external_mini_cluster.cc:1428] Setting key 7835110061af7bc70e62a654d9d4400e
I20260928 20:24:43.996838 28954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } }
I20260928 20:24:43.997028 28954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:43.997093 28954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8422b1c5a61c48a0bcc88621c6773827, State: Initialized, Role: FOLLOWER
I20260928 20:24:43.997267 28954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } }
I20260928 20:24:43.997368 28954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:43.997413 28954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:43.997468 28954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:43.998171 28954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } }
I20260928 20:24:43.998294 28954 leader_election.cc:302] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8422b1c5a61c48a0bcc88621c6773827; no voters:
I20260928 20:24:43.998517 28954 leader_election.cc:288] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:43.998543 28959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:43.998661 28959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [term 1 LEADER]: Becoming Leader. State: Replica: 8422b1c5a61c48a0bcc88621c6773827, State: Running, Role: LEADER
I20260928 20:24:43.998826 28954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:43.998816 28959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } }
I20260928 20:24:43.999295 28960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8422b1c5a61c48a0bcc88621c6773827. Latest consensus state: current_term: 1 leader_uuid: "8422b1c5a61c48a0bcc88621c6773827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } } }
I20260928 20:24:43.999392 28960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:43.999632 28961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8422b1c5a61c48a0bcc88621c6773827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8422b1c5a61c48a0bcc88621c6773827" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41831 } } }
I20260928 20:24:43.999980 28961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:44.000240 28965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:44.000789 28965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:44.002638 28965 catalog_manager.cc:1347] Generated new cluster ID: 7c02e303222e4748bac8dd7d4ae30b73
I20260928 20:24:44.002691 28965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:24:43 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:44.008145 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:43.996011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.24.186.254:41831)
0928 20:24:43.996222 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:43.996231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:43.996261 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:24:43.996832 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:43.996840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:43.996899 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:43.997068 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:43.997823 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:43.998959 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.000086 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.000741 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.000863 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.004400 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.004404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.004404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.004418 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.006406 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.006856 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.006858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.006859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.006962 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.007183 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.007185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.007185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.007440 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.007620 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.007711 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.007981 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":108,"threads_started":1}
I20260928 20:24:44.025772 28965 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:44.026551 28965 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:44.030289 28965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8422b1c5a61c48a0bcc88621c6773827: Generated new TSK 0
I20260928 20:24:44.030498 28965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:44.063583 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:44.153529 28980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:44.153743 28980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:44.153769 28980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:44.155601 28980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:44.155699 28980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:44.155764 28980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:44.158025 28980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:44.158300 28980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:44.158569 28980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:44.163714 28986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:44.163874 28980 server_base.cc:1034] running on GCE node
W20260928 20:24:44.163714 28985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:44.163796 28988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:44.164217 28980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:44.164498 28980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:44.165680 28980 hybrid_clock.cc:648] HybridClock initialized: now 1790627084165640 us; error 41 us; skew 500 ppm
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:44.168342 28980 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:44.169003 28980 webserver.cc:460] Webserver started at http://127.24.186.193:39903/ using document root <none> and password file <none>
I20260928 20:24:44.169266 28980 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:44.169337 28980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:44.169512 28980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:44.170706 28980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "d17a3a0a4ef34d3a9c19fe6e889824ca"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "1856abdba51ed6710b17d9a5ba6b7e92"
server_key_iv: "e00468e04b56ff0522c21d77a0cf298a"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.171079 28980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "d17a3a0a4ef34d3a9c19fe6e889824ca"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "1856abdba51ed6710b17d9a5ba6b7e92"
server_key_iv: "e00468e04b56ff0522c21d77a0cf298a"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.172618 28980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:44.173466 28995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.173696 28980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:44.173794 28980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "d17a3a0a4ef34d3a9c19fe6e889824ca"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "1856abdba51ed6710b17d9a5ba6b7e92"
server_key_iv: "e00468e04b56ff0522c21d77a0cf298a"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.173853 28980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:44.193215 28980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:44.195241 28980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:44.195370 28980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:44.195577 28980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:44.195972 28980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:44.196008 28980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.196035 28980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:44.196081 28980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.202368 28980 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:42471
I20260928 20:24:44.202454 29108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:42471 every 8 connection(s)
I20260928 20:24:44.202785 28980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:44.209419 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 28980
I20260928 20:24:44.209540 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:44.209743 25323 external_mini_cluster.cc:1428] Setting key 327c81f18f34fc5b213df38f904154b8
I20260928 20:24:44.213351 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.203840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:49311 (local address 127.24.186.254:41831)
0928 20:24:44.203906 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:44.203909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:44.204070 (+ 161us) server_negotiation.cc:407] Connection header received
0928 20:24:44.205053 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:44.205055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:44.205067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:44.205140 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:44.205940 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.206451 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.207469 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.207679 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.207767 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.210311 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.210312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.210313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.210326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.211925 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.212357 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.212358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.212359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.212414 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.212679 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.212679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.212680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.212801 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.212875 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.213167 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.213257 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:24:44.213562 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.203716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41831 (local address 127.24.186.193:49311)
0928 20:24:44.203931 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:44.203980 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:44.204938 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:44.205240 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:44.205244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:44.205608 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:44.205820 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.205825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.206550 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.206551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.207374 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.207377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.207779 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.207780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.207934 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.208500 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:44.208508 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:44.210217 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:44.212016 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.212024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.212026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.212266 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.212486 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.212487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.212488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.212603 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.212866 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:44.212872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:44.213019 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 20:24:44.213144 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 20:24:44.213310 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":114,"threads_started":1}
I20260928 20:24:44.214092 29109 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41831
I20260928 20:24:44.214227 29109 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:44.214453 29109 heartbeater.cc:511] Master 127.24.186.254:41831 requested a full tablet report, sending...
I20260928 20:24:44.215042 28918 ts_manager.cc:194] Registered new tserver with Master: d17a3a0a4ef34d3a9c19fe6e889824ca (127.24.186.193:42471)
I20260928 20:24:44.216418 28918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:49311
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:44.239495 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:44.326081 29114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:44.326301 29114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:44.326325 29114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:44.327404 29114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:44.327442 29114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:44.327467 29114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:44.329408 29114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:44.329751 29114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:44.329959 29114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:44.335343 29122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:44.335345 29119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:44.335345 29120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:44.335541 29114 server_base.cc:1034] running on GCE node
I20260928 20:24:44.335760 29114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:44.336035 29114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:44.337203 29114 hybrid_clock.cc:648] HybridClock initialized: now 1790627084337193 us; error 27 us; skew 500 ppm
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:44.339699 29114 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:44.340224 29114 webserver.cc:460] Webserver started at http://127.24.186.194:35247/ using document root <none> and password file <none>
I20260928 20:24:44.340456 29114 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:44.340524 29114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:44.340679 29114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:44.341753 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "ae9daebfdfd34d44a7b7850da200d166"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "642cadc425ee0cd2a7a512afd85c76af"
server_key_iv: "d804be27d5d11616ede389ade36e417b"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.342180 29114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "ae9daebfdfd34d44a7b7850da200d166"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "642cadc425ee0cd2a7a512afd85c76af"
server_key_iv: "d804be27d5d11616ede389ade36e417b"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.344267 29114 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 20:24:44.345137 29129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.345476 29114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:44.345606 29114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "ae9daebfdfd34d44a7b7850da200d166"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "642cadc425ee0cd2a7a512afd85c76af"
server_key_iv: "d804be27d5d11616ede389ade36e417b"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.345688 29114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:44.371457 29114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:44.373577 29114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:44.373759 29114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:44.374013 29114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:44.374333 29114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:44.374387 29114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.374430 29114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:44.374454 29114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.380808 29114 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:42123
I20260928 20:24:44.380848 29242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:42123 every 8 connection(s)
I20260928 20:24:44.381201 29114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:44.385565 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29114
I20260928 20:24:44.385663 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:44.385865 25323 external_mini_cluster.cc:1428] Setting key 4e0687ee0fc426f88d8f3885f2765c85
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:44.391727 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.381941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:60147 (local address 127.24.186.254:41831)
0928 20:24:44.381993 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:44.381995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:44.382226 (+ 231us) server_negotiation.cc:407] Connection header received
0928 20:24:44.383156 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:44.383157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:44.383170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:44.383227 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:44.383925 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.384442 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.385461 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.385710 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.385785 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.388579 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.388580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.388581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.388587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.390236 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.390669 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.390670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.390671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.390736 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.391022 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.391023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.391023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.391152 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.391230 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.391518 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.391636 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:24:44.391929 29245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.381886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41831 (local address 127.24.186.194:60147)
0928 20:24:44.382048 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:44.382127 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:44.383039 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:44.383308 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:44.383311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:44.383565 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:44.383755 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.383759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.384547 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.384548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.385339 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.385342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.385812 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.385813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.385967 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.386587 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:44.386597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:44.388466 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:44.390325 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.390332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.390334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.390562 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.390823 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.390824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.390824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.390952 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.391222 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:44.391228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:44.391378 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 20:24:44.391484 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 20:24:44.391654 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260928 20:24:44.392444 29243 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41831
I20260928 20:24:44.392555 29243 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:44.392762 29243 heartbeater.cc:511] Master 127.24.186.254:41831 requested a full tablet report, sending...
I20260928 20:24:44.393169 28918 ts_manager.cc:194] Registered new tserver with Master: ae9daebfdfd34d44a7b7850da200d166 (127.24.186.194:42123)
I20260928 20:24:44.393731 28918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:60147
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:44.414961 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:44.500239 29248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:44.500538 29248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:44.500579 29248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:44.501864 29248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:44.501914 29248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:44.501943 29248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:44.503566 29248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:44.503871 29248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:44.504149 29248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:44.510133 29256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:44.510257 29248 server_base.cc:1034] running on GCE node
W20260928 20:24:44.510133 29253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:44.510378 29254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:44.510586 29248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:44.510860 29248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:44.512048 29248 hybrid_clock.cc:648] HybridClock initialized: now 1790627084512038 us; error 36 us; skew 500 ppm
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:44.514746 29248 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:44.515268 29248 webserver.cc:460] Webserver started at http://127.24.186.195:34625/ using document root <none> and password file <none>
I20260928 20:24:44.515502 29248 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:44.515568 29248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:44.515722 29248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:44.518622 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "13d5ff5019eb44d8a13e73813fe3bd64"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "bda93eb1eb4f58ba70072e030ee231c2"
server_key_iv: "c9ee54243d0478562e2e1f6944360345"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.519148 29248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "13d5ff5019eb44d8a13e73813fe3bd64"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "bda93eb1eb4f58ba70072e030ee231c2"
server_key_iv: "c9ee54243d0478562e2e1f6944360345"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.521224 29248 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260928 20:24:44.522202 29263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.522585 29248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:44.522720 29248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "13d5ff5019eb44d8a13e73813fe3bd64"
format_stamp: "Formatted at 2026-09-28 20:24:44 on dist-test-slave-k477"
server_key: "bda93eb1eb4f58ba70072e030ee231c2"
server_key_iv: "c9ee54243d0478562e2e1f6944360345"
server_key_version: "encryptionkey@0"
I20260928 20:24:44.522828 29248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:44.531293 29248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:44.533595 29248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:44.533756 29248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:44.534026 29248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:44.534343 29248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:44.534395 29248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.534436 29248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:44.534462 29248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:44.540284 29248 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38775
I20260928 20:24:44.540323 29376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38775 every 8 connection(s)
I20260928 20:24:44.540697 29248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:44.550516 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29248
I20260928 20:24:44.550630 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:44.550840 25323 external_mini_cluster.cc:1428] Setting key 9783149bc16572905a2d042924c81be8
I20260928 20:24:44.551472 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.541568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:44381 (local address 127.24.186.254:41831)
0928 20:24:44.541634 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:44.541636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:44.541888 (+ 252us) server_negotiation.cc:407] Connection header received
0928 20:24:44.542919 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:44.542921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:44.542933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:44.542994 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:44.543659 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.544250 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.545229 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.545415 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.545464 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.547722 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.547724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.547724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.547731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.549183 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.549763 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.549764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.549764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.549846 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.550358 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.550360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.550360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.550522 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.550610 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.551276 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.551366 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:24:44.551713 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.541456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41831 (local address 127.24.186.195:44381)
0928 20:24:44.541701 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:44.541740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:44.542710 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:44.543066 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:44.543069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:44.543337 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:44.543555 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.543559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.544345 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.544346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.545153 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:44.545155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.545482 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.545483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:44.545585 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.546195 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:44.546203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:44.547616 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:44.549283 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.549290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.549292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.549555 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.549988 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:44.549991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:44.549992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:44.550244 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:44.550668 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:44.550678 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:44.551082 (+ 404us) client_negotiation.cc:769] Sending connection context
0928 20:24:44.551254 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 20:24:44.551465 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
I20260928 20:24:44.552248 29377 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41831
I20260928 20:24:44.552376 29377 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:44.552573 29377 heartbeater.cc:511] Master 127.24.186.254:41831 requested a full tablet report, sending...
I20260928 20:24:44.552971 28918 ts_manager.cc:194] Registered new tserver with Master: 13d5ff5019eb44d8a13e73813fe3bd64 (127.24.186.195:38775)
I20260928 20:24:44.553329 28918 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:44381
I20260928 20:24:44.562773 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:44.587435 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.579223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.24.186.193:42471)
0928 20:24:44.579377 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:44.579382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:44.579392 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:44.579417 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:44.579418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:44.579432 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:44.579512 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:44.580196 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.580922 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.581660 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.581992 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.582090 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.584509 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.584520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.584520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.584529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.585959 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.586351 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.586355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.586355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.586510 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.586779 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.586780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.586781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.586948 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.587104 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.587203 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.587299 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[28873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:44.597019 28958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:44.589055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.24.186.254:41831)
0928 20:24:44.589143 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:44.589146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:44.589158 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:44.589239 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:44.589241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:44.589254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:44.589306 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:44.590136 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.590662 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.591393 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:44.591600 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:44.591688 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:44.594622 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:44.594624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:44.594624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:44.594633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:44.595978 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.596313 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.596313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.596314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.596362 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:44.596559 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:44.596559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:44.596560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:44.596712 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:44.596785 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:44.596887 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 20:24:44.596949 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 20:24:44.597278 28918 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48846
I20260928 20:24:44.597631 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28980
I20260928 20:24:44.602849 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29114
I20260928 20:24:44.608284 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29248
I20260928 20:24:44.612890 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 28886
2026-09-28T20:24:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1940 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[29389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[29389](info): set up 2 sockets
Sep 28 20:24:44 dist-test-slave-k477 krb5kdc[29389](info): commencing operation
krb5kdc: starting...
W20260928 20:24:45.664446 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:45 dist-test-slave-k477 krb5kdc[29389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:45.783077 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44145
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:45.865955 29402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:45.866181 29402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:45.866206 29402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:45.867375 29402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:45.867415 29402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:45.867430 29402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:45.867442 29402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:45.867455 29402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:45.869091 29402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:44145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:45.869377 29402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:45.869635 29402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:45.874703 29407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:45.874819 29402 server_base.cc:1034] running on GCE node
W20260928 20:24:45.874890 29408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:45.874709 29410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:45.875192 29402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:45.875491 29402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:45.876653 29402 hybrid_clock.cc:648] HybridClock initialized: now 1790627085876646 us; error 42 us; skew 500 ppm
Sep 28 20:24:45 dist-test-slave-k477 krb5kdc[29389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627085, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:45.879302 29402 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:45.879702 29402 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:45.879781 29402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:45.879956 29402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:45.881075 29402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "f871309445244ca5b77bee182fb047cb"
format_stamp: "Formatted at 2026-09-28 20:24:45 on dist-test-slave-k477"
server_key: "963a71a6ebcdc64ce059a2d049ad92d4"
server_key_iv: "f51d24f67143ef9cc4269f8c2390e490"
server_key_version: "encryptionkey@0"
I20260928 20:24:45.881456 29402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "f871309445244ca5b77bee182fb047cb"
format_stamp: "Formatted at 2026-09-28 20:24:45 on dist-test-slave-k477"
server_key: "963a71a6ebcdc64ce059a2d049ad92d4"
server_key_iv: "f51d24f67143ef9cc4269f8c2390e490"
server_key_version: "encryptionkey@0"
I20260928 20:24:45.882829 29402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:45.883595 29416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:45.883786 29402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:45.883937 29402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "f871309445244ca5b77bee182fb047cb"
format_stamp: "Formatted at 2026-09-28 20:24:45 on dist-test-slave-k477"
server_key: "963a71a6ebcdc64ce059a2d049ad92d4"
server_key_iv: "f51d24f67143ef9cc4269f8c2390e490"
server_key_version: "encryptionkey@0"
I20260928 20:24:45.884020 29402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:45.910732 29402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:45.912654 29402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:45.912825 29402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:45.916781 29402 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:44145
I20260928 20:24:45.916854 29468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:44145 every 8 connection(s)
I20260928 20:24:45.917150 29402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:45.918114 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29402
I20260928 20:24:45.918135 29469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:45.918237 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:45.918445 25323 external_mini_cluster.cc:1428] Setting key bc105b8cc1e7ec66ca7388fa6387b8fe
I20260928 20:24:45.921775 29469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: Bootstrap starting.
I20260928 20:24:45.922477 29469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:45.922760 29469 log.cc:824] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:45.923426 29469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: No bootstrap required, opened a new log
I20260928 20:24:45.925410 29469 raft_consensus.cc:348] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } }
Sep 28 20:24:45 dist-test-slave-k477 krb5kdc[29389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:45.925582 29469 raft_consensus.cc:374] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:45.925621 29469 raft_consensus.cc:729] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f871309445244ca5b77bee182fb047cb, State: Initialized, Role: FOLLOWER
I20260928 20:24:45.925760 29469 consensus_queue.cc:260] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } }
I20260928 20:24:45.925859 29469 raft_consensus.cc:388] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:45.925894 29469 raft_consensus.cc:482] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:45.925942 29469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:45.926506 29469 raft_consensus.cc:504] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } }
I20260928 20:24:45.926617 29469 leader_election.cc:302] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f871309445244ca5b77bee182fb047cb; no voters:
I20260928 20:24:45.926800 29469 leader_election.cc:288] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:45.926828 29474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:45.926975 29474 raft_consensus.cc:686] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [term 1 LEADER]: Becoming Leader. State: Replica: f871309445244ca5b77bee182fb047cb, State: Running, Role: LEADER
I20260928 20:24:45.927071 29469 sys_catalog.cc:564] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:45.927053 29474 consensus_queue.cc:237] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } }
I20260928 20:24:45.927536 29475 sys_catalog.cc:455] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f871309445244ca5b77bee182fb047cb. Latest consensus state: current_term: 1 leader_uuid: "f871309445244ca5b77bee182fb047cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } } }
I20260928 20:24:45.927631 29475 sys_catalog.cc:458] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:45.927816 29476 sys_catalog.cc:455] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f871309445244ca5b77bee182fb047cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f871309445244ca5b77bee182fb047cb" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 44145 } } }
I20260928 20:24:45.927888 29476 sys_catalog.cc:458] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:45.928143 29483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:45.928655 29483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:45.930357 29483 catalog_manager.cc:1347] Generated new cluster ID: d65acf81c5d343df91524428b4fb46f4
I20260928 20:24:45.930420 29483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:45.930584 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:45.919036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.24.186.254:44145)
0928 20:24:45.919197 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:45.919207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:45.919238 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:24:45.919758 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:45.919766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:45.919868 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:45.919994 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:45.920773 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:45.922160 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:45.923011 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:45.923412 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:45.923533 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:45.926115 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:45.926119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:45.926119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:45.926129 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:45.928436 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:45.928850 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:45.928854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:45.928855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:45.928953 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:45.929326 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:45.929327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:45.929327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:45.929659 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:45.929826 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:45.930249 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 20:24:45.930389 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260928 20:24:45.950978 29483 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:45.951552 29483 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:45.955323 29483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: Generated new TSK 0
I20260928 20:24:45.955533 29483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:45.983242 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:46.070195 29495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:46.070438 29495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:46.070480 29495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:46.072319 29495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:46.072367 29495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:46.072402 29495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:46.075307 29495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:46.075614 29495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:46.075991 29495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:46.082413 29503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:46.082471 29500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:46.082643 29501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:46.082783 29495 server_base.cc:1034] running on GCE node
I20260928 20:24:46.083032 29495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:46.083426 29495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:46.084580 29495 hybrid_clock.cc:648] HybridClock initialized: now 1790627086084557 us; error 39 us; skew 500 ppm
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:46.087514 29495 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:46.087985 29495 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:46.088059 29495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:46.088201 29495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:46.089691 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "f28bd9afef6a4b9bb68476903c4558ff"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "cec513618a9a42d769ff45e620f61a91"
server_key_iv: "97e61dfb8f2592f3aff6c2832e030c28"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.090072 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "f28bd9afef6a4b9bb68476903c4558ff"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "cec513618a9a42d769ff45e620f61a91"
server_key_iv: "97e61dfb8f2592f3aff6c2832e030c28"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.091787 29495 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:24:46.092581 29509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.092782 29495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:46.092876 29495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "f28bd9afef6a4b9bb68476903c4558ff"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "cec513618a9a42d769ff45e620f61a91"
server_key_iv: "97e61dfb8f2592f3aff6c2832e030c28"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.092950 29495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:46.106603 29495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:46.109055 29495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:46.109212 29495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:46.109473 29495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:46.109802 29495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:46.109858 29495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.109902 29495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:46.109929 29495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.116291 29495 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35333
I20260928 20:24:46.116324 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35333 every 8 connection(s)
I20260928 20:24:46.116601 29495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:46.118798 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29495
I20260928 20:24:46.118925 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:46.119095 25323 external_mini_cluster.cc:1428] Setting key e4ef394ba0b068fd43d56fcc0adc30bb
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:46.127074 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.117253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:55497 (local address 127.24.186.254:44145)
0928 20:24:46.117317 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.117320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.117563 (+ 243us) server_negotiation.cc:407] Connection header received
0928 20:24:46.118085 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.118086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.118098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.118155 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:46.119274 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.119978 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.121041 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.121303 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.121386 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.123761 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:46.123763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:46.123764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:46.123771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:46.125584 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.126003 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.126004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.126004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.126069 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.126327 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.126327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.126328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.126513 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:46.126590 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.126852 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.126963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:24:46.127099 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.117267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44145 (local address 127.24.186.193:55497)
0928 20:24:46.117435 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.117472 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.117979 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.118233 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.118238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.118501 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:46.119188 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.119196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.120070 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.120071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.120878 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.120881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.121395 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.121396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.121557 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.122113 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:46.122124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:46.123664 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:46.125676 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.125682 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.125684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.125925 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.126140 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.126141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.126141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.126259 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.126585 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:46.126589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:46.126711 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.126802 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.126930 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 20:24:46.127943 29623 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44145
I20260928 20:24:46.128083 29623 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:46.128257 29623 heartbeater.cc:511] Master 127.24.186.254:44145 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:24:46.128859 29433 ts_manager.cc:194] Registered new tserver with Master: f28bd9afef6a4b9bb68476903c4558ff (127.24.186.193:35333)
I20260928 20:24:46.129583 29433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:55497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:46.152132 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:46.235589 29628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:46.235780 29628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:46.235819 29628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:46.236949 29628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:46.236989 29628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:46.237016 29628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:46.238515 29628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:46.238751 29628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:46.238981 29628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:46.243736 29633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:46.243830 29636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:46.243955 29628 server_base.cc:1034] running on GCE node
W20260928 20:24:46.243736 29634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:46.244330 29628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:46.244571 29628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:46.245743 29628 hybrid_clock.cc:648] HybridClock initialized: now 1790627086245727 us; error 45 us; skew 500 ppm
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:46.248263 29628 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:46.248610 29628 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:46.248682 29628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:46.248834 29628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:46.249897 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "08ae62efd8a440cfb1c1dc147b56d603"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "b29eaa2b71ce8afcead098c28e1db62f"
server_key_iv: "683aff5b310de2710c4105daec6f2712"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.250264 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "08ae62efd8a440cfb1c1dc147b56d603"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "b29eaa2b71ce8afcead098c28e1db62f"
server_key_iv: "683aff5b310de2710c4105daec6f2712"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.251559 29628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:46.252336 29642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.252504 29628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.252601 29628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "08ae62efd8a440cfb1c1dc147b56d603"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "b29eaa2b71ce8afcead098c28e1db62f"
server_key_iv: "683aff5b310de2710c4105daec6f2712"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.252683 29628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:46.268596 29628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:46.270731 29628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:46.270895 29628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:46.271132 29628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:46.271466 29628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:46.271521 29628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.271564 29628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:46.271591 29628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.277432 29628 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:35103
I20260928 20:24:46.277513 29755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:35103 every 8 connection(s)
I20260928 20:24:46.277834 29628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:46.287130 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.278540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:50255 (local address 127.24.186.254:44145)
0928 20:24:46.278583 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.278585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.278801 (+ 216us) server_negotiation.cc:407] Connection header received
0928 20:24:46.279332 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.279333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.279346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.279406 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:46.280515 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.281034 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.282025 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.282257 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.282345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.284573 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:46.284575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:46.284575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:46.284582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:46.285808 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.286230 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.286231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.286231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.286283 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.286487 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.286488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.286488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.286636 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:46.286689 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.286954 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.287036 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:46.287236 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.278529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44145 (local address 127.24.186.194:50255)
0928 20:24:46.278674 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.278709 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.279226 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.279480 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.279483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.279763 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:46.280415 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.280420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.281131 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.281132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.281931 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.281933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.282357 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.282358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.282478 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.282983 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:46.282992 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:46.284485 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:46.285883 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.285889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.285890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.286154 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.286343 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.286344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.286344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.286437 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.286697 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:46.286702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:46.286812 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.286897 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.287012 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260928 20:24:46.287761 29756 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44145
I20260928 20:24:46.287770 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29628
I20260928 20:24:46.287886 29756 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:46.287897 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:46.288039 29756 heartbeater.cc:511] Master 127.24.186.254:44145 requested a full tablet report, sending...
I20260928 20:24:46.288076 25323 external_mini_cluster.cc:1428] Setting key 98b480015be4a0d6c0fab2e8a4379c05
I20260928 20:24:46.288475 29432 ts_manager.cc:194] Registered new tserver with Master: 08ae62efd8a440cfb1c1dc147b56d603 (127.24.186.194:35103)
I20260928 20:24:46.288875 29432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:50255
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:46.312968 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:46.393736 29761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:46.393980 29761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:46.394021 29761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:46.395219 29761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:46.395288 29761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:46.395339 29761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:46.397022 29761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:44145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:46.397295 29761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:46.397526 29761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:46.402426 29769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:46.402420 29766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:46.402665 29761 server_base.cc:1034] running on GCE node
W20260928 20:24:46.402783 29767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:46.403003 29761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:46.403261 29761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:46.404445 29761 hybrid_clock.cc:648] HybridClock initialized: now 1790627086404423 us; error 43 us; skew 500 ppm
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:46.406941 29761 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:46.407341 29761 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:46.407392 29761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:46.407635 29761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:46.408782 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "8e1ac1b9e6be4779be99834b935cc80b"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "6b6148b37522df1fe4e656eec3365b95"
server_key_iv: "e58a9dfe34ce8f211169822031dfae8f"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.409076 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "8e1ac1b9e6be4779be99834b935cc80b"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "6b6148b37522df1fe4e656eec3365b95"
server_key_iv: "e58a9dfe34ce8f211169822031dfae8f"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.410288 29761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:46.410976 29775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.411209 29761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:46.411304 29761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "8e1ac1b9e6be4779be99834b935cc80b"
format_stamp: "Formatted at 2026-09-28 20:24:46 on dist-test-slave-k477"
server_key: "6b6148b37522df1fe4e656eec3365b95"
server_key_iv: "e58a9dfe34ce8f211169822031dfae8f"
server_key_version: "encryptionkey@0"
I20260928 20:24:46.411378 29761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:46.424103 29761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:46.426162 29761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:46.426327 29761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:46.426563 29761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:46.426899 29761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:46.426961 29761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.427003 29761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:46.427031 29761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:46.433305 29761 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:36773
I20260928 20:24:46.433342 29888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:36773 every 8 connection(s)
I20260928 20:24:46.433634 29761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:46.438092 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29761
I20260928 20:24:46.438197 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:46.438410 25323 external_mini_cluster.cc:1428] Setting key 414b62995f08f535cecc7cc4e91c71bf
Sep 28 20:24:46 dist-test-slave-k477 krb5kdc[29389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627086, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:46.442577 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.434432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:37749 (local address 127.24.186.254:44145)
0928 20:24:46.434502 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.434505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.434713 (+ 208us) server_negotiation.cc:407] Connection header received
0928 20:24:46.435180 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.435181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.435191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.435243 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:46.436241 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.436750 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.437702 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.437889 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.437944 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.440204 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:46.440206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:46.440206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:46.440212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:46.441396 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.441765 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.441766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.441766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.441816 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.442039 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.442040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.442040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.442153 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:46.442205 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.442446 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.442512 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:24:46.442814 29891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.434405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:44145 (local address 127.24.186.195:37749)
0928 20:24:46.434580 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.434616 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.435077 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.435299 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.435302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.435552 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:46.436146 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.436152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.436833 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.436834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.437612 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.437614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.437974 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.437975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.438123 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.438730 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:46.438738 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:46.440095 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:46.441473 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.441479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.441481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.441698 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.441881 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:46.441882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:46.441882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:46.441983 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:46.442213 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:46.442217 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:46.442333 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.442418 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.442544 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":114,"threads_started":1}
I20260928 20:24:46.443367 29889 heartbeater.cc:347] Connected to a master server at 127.24.186.254:44145
I20260928 20:24:46.443446 29889 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:46.443620 29889 heartbeater.cc:511] Master 127.24.186.254:44145 requested a full tablet report, sending...
I20260928 20:24:46.444056 29432 ts_manager.cc:194] Registered new tserver with Master: 8e1ac1b9e6be4779be99834b935cc80b (127.24.186.195:36773)
I20260928 20:24:46.444440 29432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:37749
I20260928 20:24:46.450037 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:24:46.460556 29472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.451688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.24.186.254:44145)
0928 20:24:46.451768 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.451770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.451960 (+ 190us) server_negotiation.cc:407] Connection header received
0928 20:24:46.452009 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.452010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.452019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.452081 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:46.452731 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.453859 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.455248 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.455602 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.455694 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.457431 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:46.457433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:46.457433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:46.457441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:46.459085 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.459528 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.459530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.459530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.459604 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:46.459887 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:46.459888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:46.459888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:46.460069 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:46.460175 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.460396 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.460485 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:24:46.462424 29432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:46.464030 29432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:46.470520 29900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36773 (local address 127.0.0.1:56296)
0928 20:24:46.466602 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.466613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.466665 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.466913 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.466916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.466918 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:46.466994 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.466997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468326 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.468327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.469335 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.469337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470195 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.470279 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470295 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.470363 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.470423 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260928 20:24:46.470679 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.24.186.194:35103)
0928 20:24:46.466912 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.466917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.466926 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:46.466961 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.466963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.466978 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.467059 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:46.467258 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468404 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.469741 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470291 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470370 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470421 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.470538 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.470587 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":72,"threads_started":1}
I20260928 20:24:46.470847 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35333 (local address 127.0.0.1:47454)
0928 20:24:46.466759 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.466766 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.466798 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.467356 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.467357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.467358 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:46.467434 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.467435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468415 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.468417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.469642 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.469644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470608 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.470677 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470682 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.470739 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.470774 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260928 20:24:46.470887 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56296 (local address 127.24.186.195:36773)
0928 20:24:46.466727 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.466732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.466744 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:46.466784 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.466785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.466803 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.466887 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:46.467073 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468222 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.469440 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470103 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470178 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470241 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.470758 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.470804 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260928 20:24:46.471061 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:35103 (local address 127.0.0.1:35528)
0928 20:24:46.466483 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.466529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.466601 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.467077 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.467078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.467080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:46.467156 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.467157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468503 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.468504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.469642 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.469644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470381 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.470449 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470454 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.470514 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.470557 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":31,"thread_start_us":44,"threads_started":1}
I20260928 20:24:46.471163 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.466763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.24.186.193:35333)
0928 20:24:46.467129 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.467133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.467142 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:46.467181 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.467183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.467204 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.467274 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:46.467512 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.468317 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.469750 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.470454 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.470534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.470580 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.471025 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.471070 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":95,"threads_started":1}
I20260928 20:24:46.471596 29823 tablet_service.cc:1467] Processing CreateTablet for tablet eaa8cc77078140f6ac1149af46e6e397 (DEFAULT_TABLE table=test-table [id=7ce07ebce45342f68aab39fd79eab603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:46.471736 29690 tablet_service.cc:1467] Processing CreateTablet for tablet eaa8cc77078140f6ac1149af46e6e397 (DEFAULT_TABLE table=test-table [id=7ce07ebce45342f68aab39fd79eab603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:46.471859 29557 tablet_service.cc:1467] Processing CreateTablet for tablet eaa8cc77078140f6ac1149af46e6e397 (DEFAULT_TABLE table=test-table [id=7ce07ebce45342f68aab39fd79eab603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:46.471980 29823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa8cc77078140f6ac1149af46e6e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:46.472051 29690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa8cc77078140f6ac1149af46e6e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:46.472139 29557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa8cc77078140f6ac1149af46e6e397. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:46.474454 29906 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Bootstrap starting.
I20260928 20:24:46.475129 29906 tablet_bootstrap.cc:654] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:46.475296 29908 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Bootstrap starting.
I20260928 20:24:46.475826 29906 log.cc:824] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:46.476022 29908 tablet_bootstrap.cc:654] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:46.476106 29907 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Bootstrap starting.
I20260928 20:24:46.476634 29908 log.cc:824] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:46.476691 29906 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: No bootstrap required, opened a new log
I20260928 20:24:46.476691 29907 tablet_bootstrap.cc:654] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:46.476748 29906 ts_tablet_manager.cc:1397] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:46.477270 29907 log.cc:824] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:46.477454 29908 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: No bootstrap required, opened a new log
I20260928 20:24:46.477608 29908 ts_tablet_manager.cc:1397] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:46.478830 29906 raft_consensus.cc:348] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.479013 29906 raft_consensus.cc:374] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:46.479050 29906 raft_consensus.cc:729] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ae62efd8a440cfb1c1dc147b56d603, State: Initialized, Role: FOLLOWER
I20260928 20:24:46.479162 29906 consensus_queue.cc:260] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.479465 29906 ts_tablet_manager.cc:1428] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:46.479470 29756 heartbeater.cc:503] Master 127.24.186.254:44145 was elected leader, sending a full tablet report...
I20260928 20:24:46.479594 29908 raft_consensus.cc:348] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.479869 29908 raft_consensus.cc:374] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:46.479918 29908 raft_consensus.cc:729] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1ac1b9e6be4779be99834b935cc80b, State: Initialized, Role: FOLLOWER
I20260928 20:24:46.480036 29908 consensus_queue.cc:260] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.480149 29907 tablet_bootstrap.cc:492] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: No bootstrap required, opened a new log
I20260928 20:24:46.480221 29907 ts_tablet_manager.cc:1397] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 20:24:46.480321 29908 ts_tablet_manager.cc:1428] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:46.480358 29889 heartbeater.cc:503] Master 127.24.186.254:44145 was elected leader, sending a full tablet report...
I20260928 20:24:46.481967 29907 raft_consensus.cc:348] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.482102 29907 raft_consensus.cc:374] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:46.482123 29907 raft_consensus.cc:729] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28bd9afef6a4b9bb68476903c4558ff, State: Initialized, Role: FOLLOWER
I20260928 20:24:46.482254 29907 consensus_queue.cc:260] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.482528 29907 ts_tablet_manager.cc:1428] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 20:24:46.482558 29623 heartbeater.cc:503] Master 127.24.186.254:44145 was elected leader, sending a full tablet report...
W20260928 20:24:46.528424 29757 tablet.cc:2369] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:46.617398 29624 tablet.cc:2369] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:46.684206 29890 tablet.cc:2369] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:46.830265 29912 raft_consensus.cc:482] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:46.830446 29912 raft_consensus.cc:504] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.830858 29912 leader_election.cc:288] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e1ac1b9e6be4779be99834b935cc80b (127.24.186.195:36773), f28bd9afef6a4b9bb68476903c4558ff (127.24.186.193:35333)
I20260928 20:24:46.833907 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.831228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:59953 (local address 127.24.186.193:35333)
0928 20:24:46.831276 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.831278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.831320 (+ 42us) server_negotiation.cc:407] Connection header received
0928 20:24:46.831423 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.831424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.831436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.831484 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:46.831678 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.832150 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833177 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.833534 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833589 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.833596 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.833784 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.833828 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:46.833933 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.831170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:35205 (local address 127.24.186.195:36773)
0928 20:24:46.831220 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.831223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.831283 (+ 60us) server_negotiation.cc:407] Connection header received
0928 20:24:46.831355 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.831356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.831367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.831429 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:46.831586 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.832156 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833193 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.833589 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833658 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.833666 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.833821 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.833863 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:24:46.834040 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.831106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35333 (local address 127.24.186.194:59953)
0928 20:24:46.831254 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.831265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.831334 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.831513 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.831514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.831516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:46.831602 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.831604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.832228 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.832229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.833078 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.833080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.833613 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.833702 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.833706 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.833755 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.833815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":31,"thread_start_us":75,"threads_started":1}
I20260928 20:24:46.834213 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.831050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36773 (local address 127.24.186.194:35205)
0928 20:24:46.831212 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.831224 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:46.831289 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:46.831445 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:46.831446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:46.831449 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:46.831511 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.831514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.832261 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.832262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.833076 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:46.833078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.833679 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.833680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:46.833724 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.833727 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:46.833803 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 20:24:46.833840 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260928 20:24:46.834501 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa8cc77078140f6ac1149af46e6e397" candidate_uuid: "08ae62efd8a440cfb1c1dc147b56d603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28bd9afef6a4b9bb68476903c4558ff" is_pre_election: true
I20260928 20:24:46.834507 29843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa8cc77078140f6ac1149af46e6e397" candidate_uuid: "08ae62efd8a440cfb1c1dc147b56d603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ac1b9e6be4779be99834b935cc80b" is_pre_election: true
I20260928 20:24:46.834678 29577 raft_consensus.cc:2454] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ae62efd8a440cfb1c1dc147b56d603 in term 0.
I20260928 20:24:46.834693 29843 raft_consensus.cc:2454] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ae62efd8a440cfb1c1dc147b56d603 in term 0.
I20260928 20:24:46.834950 29644 leader_election.cc:302] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ae62efd8a440cfb1c1dc147b56d603, f28bd9afef6a4b9bb68476903c4558ff; no voters:
I20260928 20:24:46.835093 29912 raft_consensus.cc:2781] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:46.835163 29912 raft_consensus.cc:482] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:46.835188 29912 raft_consensus.cc:3037] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:46.835906 29912 raft_consensus.cc:504] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.836048 29912 leader_election.cc:288] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [CANDIDATE]: Term 1 election: Requested vote from peers 8e1ac1b9e6be4779be99834b935cc80b (127.24.186.195:36773), f28bd9afef6a4b9bb68476903c4558ff (127.24.186.193:35333)
I20260928 20:24:46.836202 29843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa8cc77078140f6ac1149af46e6e397" candidate_uuid: "08ae62efd8a440cfb1c1dc147b56d603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1ac1b9e6be4779be99834b935cc80b"
I20260928 20:24:46.836278 29577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa8cc77078140f6ac1149af46e6e397" candidate_uuid: "08ae62efd8a440cfb1c1dc147b56d603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28bd9afef6a4b9bb68476903c4558ff"
I20260928 20:24:46.836282 29843 raft_consensus.cc:3037] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:46.836346 29577 raft_consensus.cc:3037] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:46.837173 29843 raft_consensus.cc:2454] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ae62efd8a440cfb1c1dc147b56d603 in term 1.
I20260928 20:24:46.837173 29577 raft_consensus.cc:2454] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ae62efd8a440cfb1c1dc147b56d603 in term 1.
I20260928 20:24:46.837360 29643 leader_election.cc:302] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ae62efd8a440cfb1c1dc147b56d603, 8e1ac1b9e6be4779be99834b935cc80b; no voters:
I20260928 20:24:46.837473 29912 raft_consensus.cc:2781] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:46.837610 29912 raft_consensus.cc:686] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 1 LEADER]: Becoming Leader. State: Replica: 08ae62efd8a440cfb1c1dc147b56d603, State: Running, Role: LEADER
I20260928 20:24:46.837806 29912 consensus_queue.cc:237] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } }
I20260928 20:24:46.838671 29433 catalog_manager.cc:5495] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08ae62efd8a440cfb1c1dc147b56d603 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "08ae62efd8a440cfb1c1dc147b56d603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ae62efd8a440cfb1c1dc147b56d603" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:46.915033 29902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.912498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.24.186.194:35103)
0928 20:24:46.912564 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:46.912566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:46.912576 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:46.912636 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:46.912637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:46.912654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:46.912723 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:46.912924 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.913495 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.914322 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:46.914591 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:46.914650 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:46.914784 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:46.914885 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:46.914940 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:46.914957 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 20:24:46.914973 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 20:24:46.916957 29843 raft_consensus.cc:1261] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 1 FOLLOWER]: Refusing update from remote peer 08ae62efd8a440cfb1c1dc147b56d603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:46.917006 29577 raft_consensus.cc:1261] T eaa8cc77078140f6ac1149af46e6e397 P f28bd9afef6a4b9bb68476903c4558ff [term 1 FOLLOWER]: Refusing update from remote peer 08ae62efd8a440cfb1c1dc147b56d603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:46.917347 29917 consensus_queue.cc:1035] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1ac1b9e6be4779be99834b935cc80b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:46.917495 29912 consensus_queue.cc:1035] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bd9afef6a4b9bb68476903c4558ff" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:46.919768 29921 mvcc.cc:205] Tried to move back new op lower bound from 7334408548008980480 to 7334408547688304640. Current Snapshot: MvccSnapshot[applied={T|T < 7334408548008980480}]
I20260928 20:24:46.922045 29432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:46.922168 29432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:46.923414 29432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f871309445244ca5b77bee182fb047cb: Sending DeleteTablet for 3 replicas of tablet eaa8cc77078140f6ac1149af46e6e397
I20260928 20:24:46.923592 29823 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa8cc77078140f6ac1149af46e6e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:46 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56296
I20260928 20:24:46.923705 29822 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa8cc77078140f6ac1149af46e6e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:46 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56296
I20260928 20:24:46.923926 29931 tablet_replica.cc:330] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: stopping tablet replica
I20260928 20:24:46.923924 29690 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa8cc77078140f6ac1149af46e6e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:46 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35528
I20260928 20:24:46.924081 29931 raft_consensus.cc:2229] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:46.924108 29932 tablet_replica.cc:330] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: stopping tablet replica
I20260928 20:24:46.924166 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29495
I20260928 20:24:46.924199 29932 raft_consensus.cc:2229] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:46.924203 29557 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa8cc77078140f6ac1149af46e6e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:46 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:47454
I20260928 20:24:46.924374 29931 raft_consensus.cc:2258] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:46.924926 29932 raft_consensus.cc:2258] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:46.925136 29931 ts_tablet_manager.cc:1905] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:46.925346 29932 ts_tablet_manager.cc:1905] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:46.927060 29931 ts_tablet_manager.cc:1918] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:46.927225 29931 log.cc:1196] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/eaa8cc77078140f6ac1149af46e6e397
I20260928 20:24:46.927209 29690 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa8cc77078140f6ac1149af46e6e397 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:46 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35528
I20260928 20:24:46.927331 29932 ts_tablet_manager.cc:1918] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:46.927404 29932 log.cc:1196] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/eaa8cc77078140f6ac1149af46e6e397
I20260928 20:24:46.927711 29932 ts_tablet_manager.cc:1939] T eaa8cc77078140f6ac1149af46e6e397 P 08ae62efd8a440cfb1c1dc147b56d603: Deleting consensus metadata
I20260928 20:24:46.927726 29931 ts_tablet_manager.cc:1939] T eaa8cc77078140f6ac1149af46e6e397 P 8e1ac1b9e6be4779be99834b935cc80b: Deleting consensus metadata
W20260928 20:24:46.928418 29418 catalog_manager.cc:4808] TS 08ae62efd8a440cfb1c1dc147b56d603 (127.24.186.194:35103): delete failed for tablet eaa8cc77078140f6ac1149af46e6e397 because the tablet was not found. No further retry: Not found: Tablet not found: eaa8cc77078140f6ac1149af46e6e397
W20260928 20:24:46.928458 29417 catalog_manager.cc:4808] TS 8e1ac1b9e6be4779be99834b935cc80b (127.24.186.195:36773): delete failed for tablet eaa8cc77078140f6ac1149af46e6e397 because the tablet was not found. No further retry: Not found: Tablet not found: eaa8cc77078140f6ac1149af46e6e397
W20260928 20:24:46.931083 29418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:35333 (error 108)
I20260928 20:24:46.931339 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29628
I20260928 20:24:46.932145 29901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:46.931456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35333 (local address 127.0.0.1:47464)
0928 20:24:46.932083 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:46.932115 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607}
W20260928 20:24:46.932273 29418 catalog_manager.cc:4584] TS f28bd9afef6a4b9bb68476903c4558ff (127.24.186.193:35333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eaa8cc77078140f6ac1149af46e6e397: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35333: connect: Connection refused (error 111)
I20260928 20:24:46.937296 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29761
I20260928 20:24:46.944058 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29402
2026-09-28T20:24:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2330 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T20:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:46Z Disabled control of system clock
I20260928 20:24:46.972081 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41165
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:24:47.054926 29937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:47.055161 29937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:47.055186 29937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:47.056433 29937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:47.056478 29937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:47.056490 29937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:47.056504 29937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:47.056515 29937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:47.057814 29937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:47.058014 29937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:47.058276 29937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:47.063076 29945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.063170 29943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.063095 29942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.064150 29937 server_base.cc:1034] running on GCE node
I20260928 20:24:47.064375 29937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:47.064633 29937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:47.065809 29937 hybrid_clock.cc:648] HybridClock initialized: now 1790627087065790 us; error 30 us; skew 500 ppm
I20260928 20:24:47.067157 29937 webserver.cc:460] Webserver started at http://127.24.186.254:33401/ using document root <none> and password file <none>
I20260928 20:24:47.067423 29937 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:47.067499 29937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:47.067654 29937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:47.068833 29937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "6cb6e3a08e064413a9f28dc7b83f30e4"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "793ce7e8aa8d1aaea325260cc1187c94"
server_key_iv: "2f43f7846f599af508effdfbd7112e36"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.069219 29937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "6cb6e3a08e064413a9f28dc7b83f30e4"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "793ce7e8aa8d1aaea325260cc1187c94"
server_key_iv: "2f43f7846f599af508effdfbd7112e36"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.070606 29937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:47.071410 29951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.071589 29937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:47.071695 29937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "6cb6e3a08e064413a9f28dc7b83f30e4"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "793ce7e8aa8d1aaea325260cc1187c94"
server_key_iv: "2f43f7846f599af508effdfbd7112e36"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.071781 29937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:47.075248 29937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:47.075543 29937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:47.075699 29937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:47.079774 29937 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41165
I20260928 20:24:47.079867 30003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41165 every 8 connection(s)
I20260928 20:24:47.080215 29937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:47.080857 30004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:47.083731 30004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: Bootstrap starting.
I20260928 20:24:47.084396 30004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:47.084709 30004 log.cc:824] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:47.085461 30004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: No bootstrap required, opened a new log
I20260928 20:24:47.086529 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 29937
I20260928 20:24:47.086656 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:47.086833 25323 external_mini_cluster.cc:1428] Setting key 5316cdc280a73084890f0c26eb3256be
I20260928 20:24:47.087247 30004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } }
I20260928 20:24:47.087460 30004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:47.087500 30004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb6e3a08e064413a9f28dc7b83f30e4, State: Initialized, Role: FOLLOWER
I20260928 20:24:47.087643 30004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } }
I20260928 20:24:47.087729 30004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:47.087770 30004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:47.087854 30004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:47.088595 30004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } }
I20260928 20:24:47.088721 30004 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb6e3a08e064413a9f28dc7b83f30e4; no voters:
I20260928 20:24:47.088930 30004 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:47.088979 30009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:47.089102 30009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb6e3a08e064413a9f28dc7b83f30e4, State: Running, Role: LEADER
I20260928 20:24:47.089201 30009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } }
I20260928 20:24:47.089243 30004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:47.089505 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.087401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.24.186.254:41165)
0928 20:24:47.087686 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.087695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.087726 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:24:47.087864 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.087872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.087918 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.088065 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.088734 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.088738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.088738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.088743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.088795 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.088806 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.088886 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.088994 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.089057 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":114,"threads_started":1}
I20260928 20:24:47.089527 30009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } } }
I20260928 20:24:47.090134 30009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:47.090287 30009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb6e3a08e064413a9f28dc7b83f30e4. Latest consensus state: current_term: 1 leader_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb6e3a08e064413a9f28dc7b83f30e4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41165 } } }
I20260928 20:24:47.090343 30019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:47.090353 30009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:47.090939 30019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:47.092648 30019 catalog_manager.cc:1347] Generated new cluster ID: 801818de27754b189e45a17e1ca155dd
I20260928 20:24:47.092703 30019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:47.109771 30019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:47.109977 30019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:47.115795 30019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: Generated new TSK 0
I20260928 20:24:47.116103 30019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:47.123669 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--builtin_ntp_servers=127.24.186.212:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:24:47.214908 30028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:47.215132 30028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:47.215157 30028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:47.216430 30028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:47.216475 30028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:47.216502 30028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:47.217861 30028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:47.218132 30028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:47.218441 30028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:47.223717 30033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.223721 30028 server_base.cc:1034] running on GCE node
W20260928 20:24:47.223990 30034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.224115 30036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.224375 30028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:47.224670 30028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:47.225888 30028 hybrid_clock.cc:648] HybridClock initialized: now 1790627087225855 us; error 44 us; skew 500 ppm
I20260928 20:24:47.227303 30028 webserver.cc:460] Webserver started at http://127.24.186.193:34969/ using document root <none> and password file <none>
I20260928 20:24:47.227576 30028 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:47.227651 30028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:47.227857 30028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:47.228978 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "a10d1fbce6ce49aeb721e3742ab59945"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "4b7b6ae64ea7172586291fd7196041c6"
server_key_iv: "84e2d9bb80bc8c489803da2bf68275e8"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.229378 30028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "a10d1fbce6ce49aeb721e3742ab59945"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "4b7b6ae64ea7172586291fd7196041c6"
server_key_iv: "84e2d9bb80bc8c489803da2bf68275e8"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.230804 30028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:47.231621 30042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.231834 30028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:47.231946 30028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "a10d1fbce6ce49aeb721e3742ab59945"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "4b7b6ae64ea7172586291fd7196041c6"
server_key_iv: "84e2d9bb80bc8c489803da2bf68275e8"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.232036 30028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:47.235466 30028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:47.235757 30028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:47.235926 30028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:47.236150 30028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:47.236483 30028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:47.236538 30028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.236578 30028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:47.236608 30028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.242257 30028 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:45251
I20260928 20:24:47.242338 30155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:45251 every 8 connection(s)
I20260928 20:24:47.242684 30028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:47.245954 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.243536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:60679 (local address 127.24.186.254:41165)
0928 20:24:47.243614 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.243615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.243928 (+ 313us) server_negotiation.cc:407] Connection header received
0928 20:24:47.244383 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.244384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.244392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.244465 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.245660 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.245661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.245662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.245666 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.245683 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.245687 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.245747 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.245878 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.245895 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:24:47.246181 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.243489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41165 (local address 127.24.186.193:60679)
0928 20:24:47.243672 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.243713 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.244277 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.244539 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.244543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.244550 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.245485 (+ 935us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.245486 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.245517 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.245548 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.245774 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.245777 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.245779 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.245837 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.245889 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 20:24:47.246745 30156 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41165
I20260928 20:24:47.246841 30156 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:47.246996 30156 heartbeater.cc:511] Master 127.24.186.254:41165 requested a full tablet report, sending...
I20260928 20:24:47.247547 29968 ts_manager.cc:194] Registered new tserver with Master: a10d1fbce6ce49aeb721e3742ab59945 (127.24.186.193:45251)
I20260928 20:24:47.249508 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30028
I20260928 20:24:47.249630 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:47.249845 25323 external_mini_cluster.cc:1428] Setting key 615140cc648d3d0fac0335fd334a6bec
I20260928 20:24:47.250777 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--builtin_ntp_servers=127.24.186.212:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 20:24:47.336751 30159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:47.336968 30159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:47.336993 30159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:47.338239 30159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:47.338284 30159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:47.338311 30159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:47.339710 30159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:47.340006 30159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:47.340319 30159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:47.345398 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.345403 30165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.345403 30164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.345422 30159 server_base.cc:1034] running on GCE node
I20260928 20:24:47.345829 30159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:47.346089 30159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:47.347258 30159 hybrid_clock.cc:648] HybridClock initialized: now 1790627087347260 us; error 49 us; skew 500 ppm
I20260928 20:24:47.348564 30159 webserver.cc:460] Webserver started at http://127.24.186.194:46263/ using document root <none> and password file <none>
I20260928 20:24:47.348820 30159 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:47.348892 30159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:47.349043 30159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:47.350265 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "30d51f7bf9c4414594900114b161b307"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "b274c8ec332a9f812f74b3323b6373fe"
server_key_iv: "749f79d5d2e70f54a1260fdae6975e7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.350658 30159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "30d51f7bf9c4414594900114b161b307"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "b274c8ec332a9f812f74b3323b6373fe"
server_key_iv: "749f79d5d2e70f54a1260fdae6975e7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.352015 30159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:47.352746 30173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.352910 30159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:47.353014 30159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "30d51f7bf9c4414594900114b161b307"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "b274c8ec332a9f812f74b3323b6373fe"
server_key_iv: "749f79d5d2e70f54a1260fdae6975e7d"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.353092 30159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:47.356582 30159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:47.356861 30159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:47.357005 30159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:47.357226 30159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:47.357527 30159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:47.357578 30159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.357614 30159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:47.357642 30159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.363415 30159 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:37353
I20260928 20:24:47.363451 30286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:37353 every 8 connection(s)
I20260928 20:24:47.363786 30159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:47.366000 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30159
I20260928 20:24:47.366132 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:47.366302 25323 external_mini_cluster.cc:1428] Setting key 985ee2c61900b5ab055e9918114959d4
I20260928 20:24:47.367066 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.364586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:58719 (local address 127.24.186.254:41165)
0928 20:24:47.364661 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.364663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.364890 (+ 227us) server_negotiation.cc:407] Connection header received
0928 20:24:47.365404 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.365405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.365411 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.365476 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.366822 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.366824 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.366824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.366828 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.366844 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.366849 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.366901 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.366985 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.367000 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:24:47.367277 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.364557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41165 (local address 127.24.186.194:58719)
0928 20:24:47.364732 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.364774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.365296 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.365539 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.365542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.365551 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.366670 (+ 1119us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.366671 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.366700 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.366729 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.366910 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.366915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.366918 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.366986 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.367048 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260928 20:24:47.367308 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--builtin_ntp_servers=127.24.186.212:43741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 20:24:47.367871 30287 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41165
I20260928 20:24:47.367985 30287 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:47.368130 30287 heartbeater.cc:511] Master 127.24.186.254:41165 requested a full tablet report, sending...
I20260928 20:24:47.368516 29968 ts_manager.cc:194] Registered new tserver with Master: 30d51f7bf9c4414594900114b161b307 (127.24.186.194:37353)
W20260928 20:24:47.451665 30290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:47.451928 30290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:47.451953 30290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:47.453184 30290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:47.453234 30290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:47.453262 30290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:47.454663 30290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:47.454945 30290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:47.455274 30290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:47.462487 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.462561 30290 server_base.cc:1034] running on GCE node
W20260928 20:24:47.462447 30298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.462471 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.462975 30290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:47.463219 30290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:47.464418 30290 hybrid_clock.cc:648] HybridClock initialized: now 1790627087464406 us; error 30 us; skew 500 ppm
I20260928 20:24:47.465811 30290 webserver.cc:460] Webserver started at http://127.24.186.195:35085/ using document root <none> and password file <none>
I20260928 20:24:47.466075 30290 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:47.466157 30290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:47.466310 30290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:47.467407 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "92c7a482b26c4c7492ee76a87f056c4e"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "aaa9cd1e4bebacaaef24bb1a688f76c8"
server_key_iv: "d6f80bfd9f3d765856e63d69eb8a42b7"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.467787 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "92c7a482b26c4c7492ee76a87f056c4e"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "aaa9cd1e4bebacaaef24bb1a688f76c8"
server_key_iv: "d6f80bfd9f3d765856e63d69eb8a42b7"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.469316 30290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:47.470175 30304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.470384 30290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:47.470489 30290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "92c7a482b26c4c7492ee76a87f056c4e"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "aaa9cd1e4bebacaaef24bb1a688f76c8"
server_key_iv: "d6f80bfd9f3d765856e63d69eb8a42b7"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.470578 30290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:47.474434 30290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:47.474725 30290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:47.475175 30290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:47.475610 30290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:47.476094 30290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:47.476155 30290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.476198 30290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:47.476226 30290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.484165 30290 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:34289
I20260928 20:24:47.484261 30417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:34289 every 8 connection(s)
I20260928 20:24:47.484840 30290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:47.488245 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.485745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:43179 (local address 127.24.186.254:41165)
0928 20:24:47.485860 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.485862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.485987 (+ 125us) server_negotiation.cc:407] Connection header received
0928 20:24:47.486497 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.486498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.486503 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.486555 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.487787 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.487788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.487792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.487856 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.487881 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.487886 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.487958 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.488159 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.488177 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 20:24:47.488498 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.485678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41165 (local address 127.24.186.195:43179)
0928 20:24:47.485884 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.485921 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.486404 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.486611 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.486614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.486623 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.487601 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.487602 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.487632 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.487662 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.487978 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.487986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.487990 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.488118 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.488210 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 20:24:47.489441 30418 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41165
I20260928 20:24:47.489575 30418 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:47.489792 30418 heartbeater.cc:511] Master 127.24.186.254:41165 requested a full tablet report, sending...
I20260928 20:24:47.490396 29968 ts_manager.cc:194] Registered new tserver with Master: 92c7a482b26c4c7492ee76a87f056c4e (127.24.186.195:34289)
I20260928 20:24:47.493016 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30290
I20260928 20:24:47.493112 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:47.493428 25323 external_mini_cluster.cc:1428] Setting key 8083e73461c18680c50e913042a55ce2
I20260928 20:24:47.494316 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:24:47.497267 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.495663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.24.186.254:41165)
0928 20:24:47.495754 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.495756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.495943 (+ 187us) server_negotiation.cc:407] Connection header received
0928 20:24:47.496007 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.496008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.496017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.496098 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.496870 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.496872 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.496873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.496877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.496914 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.496919 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.497014 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.497180 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.497203 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 20:24:47.499413 29968 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:47.501084 29968 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:47.505159 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:34289 (local address 127.0.0.1:59014)
0928 20:24:47.503387 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.503431 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.503484 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.504089 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.504090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.504091 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.504688 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.504689 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.504705 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.504733 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.504955 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.504963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.504966 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.505045 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.505060 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":120,"threads_started":1}
I20260928 20:24:47.505330 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59014 (local address 127.24.186.195:34289)
0928 20:24:47.503920 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.503925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.503941 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:24:47.503963 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.503965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.503971 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.504287 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.504824 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.504831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.504831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.504835 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.504874 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.504896 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.504957 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.505228 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.505244 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":100,"threads_started":1}
I20260928 20:24:47.505275 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:37353 (local address 127.0.0.1:46958)
0928 20:24:47.503578 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.503588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.503628 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.504218 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.504219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.504222 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.504839 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.504840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.504845 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.504853 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.505089 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.505090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.505091 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.505152 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.505166 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":109,"threads_started":1}
I20260928 20:24:47.505451 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.24.186.193:45251)
0928 20:24:47.503911 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.503914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.503923 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:47.503948 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.503950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.503958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.504031 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.505053 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.505061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.505061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.505066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.505102 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.505130 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.505191 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.505327 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.505346 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260928 20:24:47.505518 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45251 (local address 127.0.0.1:53968)
0928 20:24:47.503633 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.503645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.503698 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.504056 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.504063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.504070 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.504970 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.504970 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.504978 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.504990 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.505187 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.505188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.505189 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.505241 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.505254 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260928 20:24:47.505669 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.503332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.24.186.194:37353)
0928 20:24:47.503729 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.503734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.503745 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:47.503771 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.503773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.503782 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.503884 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.504936 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.504944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.504945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.504950 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.504982 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.505011 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.505062 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.505563 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.505582 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":81,"threads_started":1}
I20260928 20:24:47.506055 30352 tablet_service.cc:1467] Processing CreateTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce (DEFAULT_TABLE table=test-table [id=e9ac2f236b3e4b56a8aec548b21b0fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:47.506121 30089 tablet_service.cc:1467] Processing CreateTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce (DEFAULT_TABLE table=test-table [id=e9ac2f236b3e4b56a8aec548b21b0fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:47.506394 30089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c38bf2529c4ab7841f2f4aa39b07ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:47.506300 30221 tablet_service.cc:1467] Processing CreateTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce (DEFAULT_TABLE table=test-table [id=e9ac2f236b3e4b56a8aec548b21b0fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:47.506413 30352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c38bf2529c4ab7841f2f4aa39b07ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:47.506593 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c38bf2529c4ab7841f2f4aa39b07ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:47.509559 30432 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: Bootstrap starting.
I20260928 20:24:47.509753 30433 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Bootstrap starting.
I20260928 20:24:47.509986 30434 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Bootstrap starting.
I20260928 20:24:47.510342 30432 tablet_bootstrap.cc:654] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:47.510589 30434 tablet_bootstrap.cc:654] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:47.510689 30432 log.cc:824] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:47.510818 30433 tablet_bootstrap.cc:654] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:47.510926 30434 log.cc:824] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:47.511104 30433 log.cc:824] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:47.511595 30432 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: No bootstrap required, opened a new log
I20260928 20:24:47.511669 30434 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: No bootstrap required, opened a new log
I20260928 20:24:47.511672 30432 ts_tablet_manager.cc:1397] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:47.511730 30434 ts_tablet_manager.cc:1397] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:47.514430 30432 raft_consensus.cc:348] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.514430 30434 raft_consensus.cc:348] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.514631 30432 raft_consensus.cc:374] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:47.514631 30434 raft_consensus.cc:374] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:47.514667 30432 raft_consensus.cc:729] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10d1fbce6ce49aeb721e3742ab59945, State: Initialized, Role: FOLLOWER
I20260928 20:24:47.514667 30434 raft_consensus.cc:729] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d51f7bf9c4414594900114b161b307, State: Initialized, Role: FOLLOWER
I20260928 20:24:47.514817 30434 consensus_queue.cc:260] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.514804 30432 consensus_queue.cc:260] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.515098 30434 ts_tablet_manager.cc:1428] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:24:47.515098 30432 ts_tablet_manager.cc:1428] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 20:24:47.515287 30287 heartbeater.cc:503] Master 127.24.186.254:41165 was elected leader, sending a full tablet report...
I20260928 20:24:47.515349 30156 heartbeater.cc:503] Master 127.24.186.254:41165 was elected leader, sending a full tablet report...
I20260928 20:24:47.516103 30433 tablet_bootstrap.cc:492] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: No bootstrap required, opened a new log
I20260928 20:24:47.516233 30433 ts_tablet_manager.cc:1397] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20260928 20:24:47.518227 30433 raft_consensus.cc:348] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.518402 30433 raft_consensus.cc:374] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:47.518452 30433 raft_consensus.cc:729] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c7a482b26c4c7492ee76a87f056c4e, State: Initialized, Role: FOLLOWER
I20260928 20:24:47.518576 30433 consensus_queue.cc:260] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.518890 30433 ts_tablet_manager.cc:1428] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:24:47.518987 30418 heartbeater.cc:503] Master 127.24.186.254:41165 was elected leader, sending a full tablet report...
W20260928 20:24:47.614436 30288 tablet.cc:2369] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:47.678236 30440 raft_consensus.cc:482] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:47.678364 30440 raft_consensus.cc:504] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.678666 30440 leader_election.cc:288] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30d51f7bf9c4414594900114b161b307 (127.24.186.194:37353), a10d1fbce6ce49aeb721e3742ab59945 (127.24.186.193:45251)
I20260928 20:24:47.686937 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.678975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:60533 (local address 127.24.186.194:37353)
0928 20:24:47.679030 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.679032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.679046 (+ 14us) server_negotiation.cc:407] Connection header received
0928 20:24:47.679079 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.679080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.679087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.679172 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.686678 (+ 7506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.686680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.686680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.686685 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.686707 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.686711 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.686792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.686851 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.686866 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:24:47.686939 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.679155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:37379 (local address 127.24.186.193:45251)
0928 20:24:47.679205 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.679207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.679251 (+ 44us) server_negotiation.cc:407] Connection header received
0928 20:24:47.679321 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.679322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.679327 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.679388 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.686657 (+ 7269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.686659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.686659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.686666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.686686 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.686705 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.686780 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.686841 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.686856 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:24:47.686939 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.678892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:37353 (local address 127.24.186.195:60533)
0928 20:24:47.678948 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.678961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.679004 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.679194 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.679195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.679197 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.686542 (+ 7345us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.686543 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.686550 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.686564 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.686788 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.686789 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.686789 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.686848 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.686863 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:24:47.686959 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.679019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45251 (local address 127.24.186.195:37379)
0928 20:24:47.679196 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.679207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.679262 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.679399 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.679400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.679402 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:47.686490 (+ 7088us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:24:47.686491 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:47.686502 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:24:47.686524 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:47.686777 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:47.686778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:47.686779 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 20:24:47.686833 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 20:24:47.686851 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":114,"threads_started":1}
I20260928 20:24:47.687526 30110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c38bf2529c4ab7841f2f4aa39b07ce" candidate_uuid: "92c7a482b26c4c7492ee76a87f056c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10d1fbce6ce49aeb721e3742ab59945" is_pre_election: true
I20260928 20:24:47.687546 30241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c38bf2529c4ab7841f2f4aa39b07ce" candidate_uuid: "92c7a482b26c4c7492ee76a87f056c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d51f7bf9c4414594900114b161b307" is_pre_election: true
I20260928 20:24:47.687697 30110 raft_consensus.cc:2454] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c7a482b26c4c7492ee76a87f056c4e in term 0.
I20260928 20:24:47.687703 30241 raft_consensus.cc:2454] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c7a482b26c4c7492ee76a87f056c4e in term 0.
I20260928 20:24:47.687949 30306 leader_election.cc:302] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c7a482b26c4c7492ee76a87f056c4e, a10d1fbce6ce49aeb721e3742ab59945; no voters:
I20260928 20:24:47.688129 30440 raft_consensus.cc:2781] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:47.688215 30440 raft_consensus.cc:482] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:47.688251 30440 raft_consensus.cc:3037] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:47.689029 30440 raft_consensus.cc:504] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.689220 30440 leader_election.cc:288] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [CANDIDATE]: Term 1 election: Requested vote from peers 30d51f7bf9c4414594900114b161b307 (127.24.186.194:37353), a10d1fbce6ce49aeb721e3742ab59945 (127.24.186.193:45251)
I20260928 20:24:47.689450 30241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c38bf2529c4ab7841f2f4aa39b07ce" candidate_uuid: "92c7a482b26c4c7492ee76a87f056c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30d51f7bf9c4414594900114b161b307"
I20260928 20:24:47.689476 30110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c38bf2529c4ab7841f2f4aa39b07ce" candidate_uuid: "92c7a482b26c4c7492ee76a87f056c4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a10d1fbce6ce49aeb721e3742ab59945"
I20260928 20:24:47.689546 30241 raft_consensus.cc:3037] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:47.689546 30110 raft_consensus.cc:3037] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:47.690515 30110 raft_consensus.cc:2454] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c7a482b26c4c7492ee76a87f056c4e in term 1.
I20260928 20:24:47.690515 30241 raft_consensus.cc:2454] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c7a482b26c4c7492ee76a87f056c4e in term 1.
I20260928 20:24:47.690724 30306 leader_election.cc:302] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30d51f7bf9c4414594900114b161b307, 92c7a482b26c4c7492ee76a87f056c4e; no voters:
I20260928 20:24:47.690838 30440 raft_consensus.cc:2781] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:47.691035 30440 raft_consensus.cc:686] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 1 LEADER]: Becoming Leader. State: Replica: 92c7a482b26c4c7492ee76a87f056c4e, State: Running, Role: LEADER
I20260928 20:24:47.691154 30440 consensus_queue.cc:237] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } }
I20260928 20:24:47.692066 29967 catalog_manager.cc:5495] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 92c7a482b26c4c7492ee76a87f056c4e (127.24.186.195). New cstate: current_term: 1 leader_uuid: "92c7a482b26c4c7492ee76a87f056c4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92c7a482b26c4c7492ee76a87f056c4e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:47.734165 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.733167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.24.186.195:34289)
0928 20:24:47.733235 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.733237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.733258 (+ 21us) server_negotiation.cc:407] Connection header received
0928 20:24:47.733304 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.733305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.733317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.733373 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.733934 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.733935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.733935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.733940 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.733954 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.733958 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.734015 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.734073 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.734088 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 20:24:47.735446 30419 tablet.cc:2369] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:47.736019 30110 raft_consensus.cc:1261] T 80c38bf2529c4ab7841f2f4aa39b07ce P a10d1fbce6ce49aeb721e3742ab59945 [term 1 FOLLOWER]: Refusing update from remote peer 92c7a482b26c4c7492ee76a87f056c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:47.736033 30241 raft_consensus.cc:1261] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 1 FOLLOWER]: Refusing update from remote peer 92c7a482b26c4c7492ee76a87f056c4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:47.736384 30442 consensus_queue.cc:1035] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30d51f7bf9c4414594900114b161b307" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:47.736515 30440 consensus_queue.cc:1035] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a10d1fbce6ce49aeb721e3742ab59945" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:47.739084 30446 mvcc.cc:205] Tried to move back new op lower bound from 7334408551364079616 to 7334408551183826944. Current Snapshot: MvccSnapshot[applied={T|T < 7334408551364079616}]
I20260928 20:24:47.741629 29967 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:47.741734 29967 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:47.742539 29967 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cb6e3a08e064413a9f28dc7b83f30e4: Sending DeleteTablet for 3 replicas of tablet 80c38bf2529c4ab7841f2f4aa39b07ce
I20260928 20:24:47.742913 30221 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:47 UTC) from {username='slave'} at 127.0.0.1:46958
I20260928 20:24:47.742944 30352 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:47 UTC) from {username='slave'} at 127.0.0.1:59014
I20260928 20:24:47.743146 30089 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c38bf2529c4ab7841f2f4aa39b07ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:47 UTC) from {username='slave'} at 127.0.0.1:53968
I20260928 20:24:47.743193 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30028
I20260928 20:24:47.744000 30455 tablet_replica.cc:330] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: stopping tablet replica
I20260928 20:24:47.744156 30455 raft_consensus.cc:2229] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:47.744369 30455 raft_consensus.cc:2258] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:47.744534 30456 tablet_replica.cc:330] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: stopping tablet replica
I20260928 20:24:47.744660 30456 raft_consensus.cc:2229] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:47.744771 30456 raft_consensus.cc:2258] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:47.745236 30456 ts_tablet_manager.cc:1905] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:47.745503 30455 ts_tablet_manager.cc:1905] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:47.746984 30456 ts_tablet_manager.cc:1918] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:47.747077 30456 log.cc:1196] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/80c38bf2529c4ab7841f2f4aa39b07ce
I20260928 20:24:47.747357 30456 ts_tablet_manager.cc:1939] T 80c38bf2529c4ab7841f2f4aa39b07ce P 30d51f7bf9c4414594900114b161b307: Deleting consensus metadata
I20260928 20:24:47.747370 30455 ts_tablet_manager.cc:1918] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:47.747546 30455 log.cc:1196] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/80c38bf2529c4ab7841f2f4aa39b07ce
I20260928 20:24:47.747818 29953 catalog_manager.cc:4841] TS 30d51f7bf9c4414594900114b161b307 (127.24.186.194:37353): tablet 80c38bf2529c4ab7841f2f4aa39b07ce (table test-table [id=e9ac2f236b3e4b56a8aec548b21b0fe1]) successfully deleted
I20260928 20:24:47.748018 30455 ts_tablet_manager.cc:1939] T 80c38bf2529c4ab7841f2f4aa39b07ce P 92c7a482b26c4c7492ee76a87f056c4e: Deleting consensus metadata
I20260928 20:24:47.749171 29955 catalog_manager.cc:4841] TS 92c7a482b26c4c7492ee76a87f056c4e (127.24.186.195:34289): tablet 80c38bf2529c4ab7841f2f4aa39b07ce (table test-table [id=e9ac2f236b3e4b56a8aec548b21b0fe1]) successfully deleted
W20260928 20:24:47.750896 29953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:45251 (error 108)
I20260928 20:24:47.751405 30427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.751272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45251 (local address 127.0.0.1:53978)
0928 20:24:47.751327 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.751368 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:24:47.751464 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30159
W20260928 20:24:47.752880 29953 catalog_manager.cc:4584] TS a10d1fbce6ce49aeb721e3742ab59945 (127.24.186.193:45251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80c38bf2529c4ab7841f2f4aa39b07ce: Network error: Client connection negotiation failed: client connection to 127.24.186.193:45251: connect: Connection refused (error 111)
I20260928 20:24:47.756755 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30290
I20260928 20:24:47.763198 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 29937
2026-09-28T20:24:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (818 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 20:24:47.783188 25323 mini_oidc.cc:152] Starting JWKS server
I20260928 20:24:47.784576 25323 webserver.cc:460] Webserver started at https://127.0.0.1:45383/ using document root <none> and password file <none>
I20260928 20:24:47.784698 25323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:24:47.784898 25323 webserver.cc:460] Webserver started at http://127.0.0.1:33023/ using document root <none> and password file <none>
2026-09-28T20:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:47Z Disabled control of system clock
I20260928 20:24:47.793011 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38425
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45383/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/testchainca.cert with env {}
W20260928 20:24:47.877034 30463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:47.877283 30463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:47.877310 30463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:47.878482 30463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:47.878526 30463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:47.878539 30463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:47.878552 30463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:47.878564 30463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:24:47.878577 30463 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:24:47.878588 30463 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45383/jwks.json
I20260928 20:24:47.880254 30463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45383/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:47.880568 30463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:47.880844 30463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:47.886098 30468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:47.886096 30469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.886334 30463 server_base.cc:1034] running on GCE node
W20260928 20:24:47.886096 30471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:47.886665 30463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:47.886931 30463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:47.888067 30463 hybrid_clock.cc:648] HybridClock initialized: now 1790627087888042 us; error 41 us; skew 500 ppm
I20260928 20:24:47.889321 30463 webserver.cc:460] Webserver started at http://127.24.186.254:36501/ using document root <none> and password file <none>
I20260928 20:24:47.889583 30463 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:47.889632 30463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:47.889794 30463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:47.890928 30463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "6a017eb5b3064260ad74221603cc4c8d"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "31b5686a054ebdb7d1c6f0fd64df8343"
server_key_iv: "ba1c144ef3ce406b29bb0a23cd7d839a"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.891297 30463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "6a017eb5b3064260ad74221603cc4c8d"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "31b5686a054ebdb7d1c6f0fd64df8343"
server_key_iv: "ba1c144ef3ce406b29bb0a23cd7d839a"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.892689 30463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:47.893414 30477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:47.893594 30463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:24:47.893688 30463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "6a017eb5b3064260ad74221603cc4c8d"
format_stamp: "Formatted at 2026-09-28 20:24:47 on dist-test-slave-k477"
server_key: "31b5686a054ebdb7d1c6f0fd64df8343"
server_key_iv: "ba1c144ef3ce406b29bb0a23cd7d839a"
server_key_version: "encryptionkey@0"
I20260928 20:24:47.893764 30463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:47.917086 30463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:47.917450 30463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:47.917568 30463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:47.921869 30463 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38425
I20260928 20:24:47.921905 30529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38425 every 8 connection(s)
I20260928 20:24:47.922268 30463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:47.923213 30530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:47.925642 30530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d: Bootstrap starting.
I20260928 20:24:47.926198 30530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:47.926434 30530 log.cc:824] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:47.927047 30530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d: No bootstrap required, opened a new log
I20260928 20:24:47.927940 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30463
I20260928 20:24:47.928049 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:47.928241 25323 external_mini_cluster.cc:1428] Setting key 1b9f42402f64979dfbecdad74ef5a969
I20260928 20:24:47.929163 30530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } }
I20260928 20:24:47.929342 30530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:47.929378 30530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a017eb5b3064260ad74221603cc4c8d, State: Initialized, Role: FOLLOWER
I20260928 20:24:47.929514 30530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } }
I20260928 20:24:47.929592 30530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:47.929615 30530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:47.929682 30530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:47.930203 30530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } }
I20260928 20:24:47.930287 30530 leader_election.cc:302] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a017eb5b3064260ad74221603cc4c8d; no voters:
I20260928 20:24:47.930460 30530 leader_election.cc:288] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:47.930506 30535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:47.930630 30535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [term 1 LEADER]: Becoming Leader. State: Replica: 6a017eb5b3064260ad74221603cc4c8d, State: Running, Role: LEADER
I20260928 20:24:47.930732 30530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:47.930744 30535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } }
I20260928 20:24:47.931180 30535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a017eb5b3064260ad74221603cc4c8d. Latest consensus state: current_term: 1 leader_uuid: "6a017eb5b3064260ad74221603cc4c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } } }
I20260928 20:24:47.931264 30535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:47.931425 30536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a017eb5b3064260ad74221603cc4c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a017eb5b3064260ad74221603cc4c8d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38425 } } }
I20260928 20:24:47.931485 30536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:47.931578 30543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:47.932073 30543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:47.934401 30543 catalog_manager.cc:1347] Generated new cluster ID: c2bb1f4e73874a0daf8dee370a550687
I20260928 20:24:47.934460 30543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:47.934463 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.928747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.24.186.254:38425)
0928 20:24:47.928944 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.928951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.928979 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:24:47.929124 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.929129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.929186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.929292 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:47.929538 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.931126 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.932339 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.932932 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.933061 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:47.933748 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:47.933751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:47.933751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:47.933917 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:47.933949 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:47.933960 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:47.934013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:47.934148 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:24:47.934249 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 20:24:47.935949 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:47.960345 30543 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:47.961149 30543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:47.966080 30543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a017eb5b3064260ad74221603cc4c8d: Generated new TSK 0
I20260928 20:24:47.966288 30543 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 20:24:47.991444 30560 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 20:24:47.991552 30560 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 20:24:47.991846 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:47.976157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.24.186.254:38425)
0928 20:24:47.976202 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:47.976204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:47.976299 (+ 95us) server_negotiation.cc:407] Connection header received
0928 20:24:47.976410 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:47.976410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:47.976421 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:47.976478 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:24:47.976637 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.977116 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.978249 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.978427 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.978479 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:47.978697 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:24:47.991556 (+ 12859us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 20:24:47.991745 (+ 189us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:24:47.991901 30534 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 20:24:47.992187 30559 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:47.976033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38425 (local address 127.0.0.1:45790)
0928 20:24:47.976220 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:47.976232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:47.976311 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:47.976497 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:47.976498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:47.976501 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:24:47.976568 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:47.976570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.977209 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.977210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:47.978161 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:47.978163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:47.978501 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:47.978502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:47.978568 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:47.978590 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:24:47.991738 (+ 13148us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 20:24:47.991872 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.186.254:38425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.24.186.254:38425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45383/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 20:24:47.992660 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30463
2026-09-28T20:24:48Z chronyd exiting
I20260928 20:24:48.029109 25323 test_util.cc:175] -----------------------------------------------
I20260928 20:24:48.029189 25323 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790627060930553-25323-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (247 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 20:24:48.030035 25323 mini_oidc.cc:152] Starting JWKS server
I20260928 20:24:48.030781 25323 webserver.cc:460] Webserver started at https://127.0.0.1:38493/ using document root <none> and password file <none>
I20260928 20:24:48.030820 25323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:24:48.031028 25323 webserver.cc:460] Webserver started at http://127.0.0.1:40261/ using document root <none> and password file <none>
2026-09-28T20:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:48Z Disabled control of system clock
I20260928 20:24:48.038931 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:43229
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:43229
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38493/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/testchainca.cert with env {}
W20260928 20:24:48.117883 30566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:48.118145 30566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:48.118183 30566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:48.119402 30566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:48.119477 30566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:48.119498 30566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:48.119518 30566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:48.119552 30566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:24:48.119575 30566 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:24:48.119592 30566 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38493/jwks.json
I20260928 20:24:48.121200 30566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:43229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38493/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:48.121497 30566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:48.121721 30566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:48.126425 30571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:48.126441 30574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:48.126435 30572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:48.126753 30566 server_base.cc:1034] running on GCE node
I20260928 20:24:48.126974 30566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:48.127220 30566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:48.128367 30566 hybrid_clock.cc:648] HybridClock initialized: now 1790627088128351 us; error 28 us; skew 500 ppm
I20260928 20:24:48.129635 30566 webserver.cc:460] Webserver started at http://127.24.186.254:42423/ using document root <none> and password file <none>
I20260928 20:24:48.129833 30566 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:48.129874 30566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:48.129958 30566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:48.130967 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "4ec4ac9238fbc209d1fb61bc6ad4ad3d"
server_key_iv: "d84af1503035be76d782320f0d16284b"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.131275 30566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "4ec4ac9238fbc209d1fb61bc6ad4ad3d"
server_key_iv: "d84af1503035be76d782320f0d16284b"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.132627 30566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:48.133301 30580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:48.133471 30566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:48.133575 30566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "4ec4ac9238fbc209d1fb61bc6ad4ad3d"
server_key_iv: "d84af1503035be76d782320f0d16284b"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.133652 30566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:48.154474 30566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:48.155707 30566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:48.156080 30566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:48.160954 30566 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:43229
I20260928 20:24:48.160998 30632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:43229 every 8 connection(s)
I20260928 20:24:48.161363 30566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:48.162019 30633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:48.163308 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30566
I20260928 20:24:48.163410 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:48.163599 25323 external_mini_cluster.cc:1428] Setting key 64ee86b812d1e823fbd14b9640fe8717
I20260928 20:24:48.165616 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: Bootstrap starting.
I20260928 20:24:48.166254 30633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:48.166566 30633 log.cc:824] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:48.167315 30633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: No bootstrap required, opened a new log
I20260928 20:24:48.169241 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.164176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.24.186.254:43229)
0928 20:24:48.164374 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.164389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.164413 (+ 24us) server_negotiation.cc:407] Connection header received
0928 20:24:48.164846 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.164851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.164913 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.165024 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:48.165251 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.166178 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.167252 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.167691 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.167786 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.168508 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:48.168512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:48.168512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:48.168682 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:48.168714 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:48.168725 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:48.168795 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:48.168891 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 20:24:48.169008 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260928 20:24:48.169610 30633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } }
I20260928 20:24:48.169771 30633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:48.169796 30633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac3ef96a68f4fe5a620fbe01672b1a7, State: Initialized, Role: FOLLOWER
I20260928 20:24:48.169979 30633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } }
I20260928 20:24:48.170089 30633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:48.170135 30633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:48.170176 30633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:48.170468 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:48.170907 30633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } }
I20260928 20:24:48.171027 30633 leader_election.cc:302] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ac3ef96a68f4fe5a620fbe01672b1a7; no voters:
I20260928 20:24:48.171204 30633 leader_election.cc:288] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:48.171304 30638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:48.171445 30633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:48.171554 30638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac3ef96a68f4fe5a620fbe01672b1a7, State: Running, Role: LEADER
I20260928 20:24:48.171703 30638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } }
I20260928 20:24:48.172608 30638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ac3ef96a68f4fe5a620fbe01672b1a7. Latest consensus state: current_term: 1 leader_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } } }
I20260928 20:24:48.172755 30638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:48.172851 30639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3ef96a68f4fe5a620fbe01672b1a7" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43229 } } }
I20260928 20:24:48.172935 30639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7 [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:48.173339 30652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:48.173396 30652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:48.173436 30653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:48.173569 30653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:48.175241 30653 catalog_manager.cc:1347] Generated new cluster ID: 4486cee3f0ac4223a3c51329cda3b703
I20260928 20:24:48.175309 30653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:48.192495 30653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:48.193287 30653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:48.205976 30653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ac3ef96a68f4fe5a620fbe01672b1a7: Generated new TSK 0
I20260928 20:24:48.206204 30653 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 20:24:48.239217 30663 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 20:24:48.239312 30663 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 20:24:48.239665 30636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.219872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.24.186.254:43229)
0928 20:24:48.219926 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.219928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.220042 (+ 114us) server_negotiation.cc:407] Connection header received
0928 20:24:48.220128 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.220129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.220149 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.220229 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:24:48.220395 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.220905 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.221716 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.221915 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.222016 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.222266 (+ 250us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:24:48.239413 (+ 17147us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38493/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 20:24:48.239593 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38493/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 20:24:48.239734 30662 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:48.219780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:43229 (local address 127.0.0.1:58196)
0928 20:24:48.219964 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:48.219973 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:48.220056 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:48.220227 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:48.220228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:48.220230 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:24:48.220294 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.220296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.221007 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.221008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.221611 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.221612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.222015 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.222016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.222089 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.222095 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:24:48.239538 (+ 17443us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38493/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 20:24:48.239636 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.186.254:43229: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38493/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":120,"threads_started":1}
W20260928 20:24:48.239733 30636 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38493/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 20:24:48.240078 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30566
2026-09-28T20:24:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 20:24:48.278100 25323 mini_oidc.cc:152] Starting JWKS server
I20260928 20:24:48.278817 25323 webserver.cc:460] Webserver started at https://127.0.0.1:39903/ using document root <none> and password file <none>
I20260928 20:24:48.278858 25323 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 20:24:48.279058 25323 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
2026-09-28T20:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:48Z Disabled control of system clock
I20260928 20:24:48.287041 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33361
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33361
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39903/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 20:24:48.368750 30669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:48.368989 30669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:48.369014 30669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:48.370203 30669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:48.370249 30669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:48.370263 30669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:48.370276 30669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:48.370288 30669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:24:48.370301 30669 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:24:48.370312 30669 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39903/jwks.json
I20260928 20:24:48.371670 30669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39903/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:48.371975 30669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:48.372260 30669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:48.377140 30677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:48.377192 30674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:48.377233 30669 server_base.cc:1034] running on GCE node
W20260928 20:24:48.377142 30675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:48.377545 30669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:48.377854 30669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:48.379029 30669 hybrid_clock.cc:648] HybridClock initialized: now 1790627088379023 us; error 38 us; skew 500 ppm
I20260928 20:24:48.380465 30669 webserver.cc:460] Webserver started at http://127.24.186.254:44027/ using document root <none> and password file <none>
I20260928 20:24:48.380728 30669 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:48.380803 30669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:48.380954 30669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:48.382056 30669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "4d5f95e7b25242f189d981c836dedc4b"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "2365314b1f0bb558f3d02ed15d9020f7"
server_key_iv: "91ec40559643948ce1368e643dea4f24"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.382440 30669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "4d5f95e7b25242f189d981c836dedc4b"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "2365314b1f0bb558f3d02ed15d9020f7"
server_key_iv: "91ec40559643948ce1368e643dea4f24"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.383963 30669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 20:24:48.385059 30683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:48.385380 30669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:48.385496 30669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "4d5f95e7b25242f189d981c836dedc4b"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "2365314b1f0bb558f3d02ed15d9020f7"
server_key_iv: "91ec40559643948ce1368e643dea4f24"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.385592 30669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:48.401261 30669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:48.401615 30669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:48.401788 30669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:48.405988 30669 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:33361
I20260928 20:24:48.406030 30735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:33361 every 8 connection(s)
I20260928 20:24:48.406390 30669 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:48.407096 30736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:48.409986 30736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b: Bootstrap starting.
I20260928 20:24:48.410595 30736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:48.410892 30736 log.cc:824] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:48.411680 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30669
I20260928 20:24:48.411789 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:48.411919 30736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b: No bootstrap required, opened a new log
I20260928 20:24:48.412040 25323 external_mini_cluster.cc:1428] Setting key 094f1b6135219f72d9fa04fb77ba0add
I20260928 20:24:48.414666 30736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } }
I20260928 20:24:48.414839 30736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:48.414883 30736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5f95e7b25242f189d981c836dedc4b, State: Initialized, Role: FOLLOWER
I20260928 20:24:48.415009 30736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } }
I20260928 20:24:48.415113 30736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:48.415170 30736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:48.415222 30736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:48.415913 30736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } }
I20260928 20:24:48.416028 30736 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5f95e7b25242f189d981c836dedc4b; no voters:
I20260928 20:24:48.416244 30736 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:48.416265 30741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:48.416376 30741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [term 1 LEADER]: Becoming Leader. State: Replica: 4d5f95e7b25242f189d981c836dedc4b, State: Running, Role: LEADER
I20260928 20:24:48.416460 30741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } }
I20260928 20:24:48.416518 30736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:48.416802 30743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5f95e7b25242f189d981c836dedc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } } }
I20260928 20:24:48.416901 30743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:48.417074 30742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5f95e7b25242f189d981c836dedc4b. Latest consensus state: current_term: 1 leader_uuid: "4d5f95e7b25242f189d981c836dedc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5f95e7b25242f189d981c836dedc4b" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33361 } } }
I20260928 20:24:48.417157 30742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:48.417639 30747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:48.418183 30747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:48.418924 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.412577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.24.186.254:33361)
0928 20:24:48.412815 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.412823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.412849 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:24:48.413368 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.413376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.413436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.413552 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:48.413904 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.414782 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.415767 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.416410 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.417264 (+ 854us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.417421 (+ 157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:48.417424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:48.417425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:48.418027 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:48.418068 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:48.418082 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:48.418193 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:48.418472 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 20:24:48.418567 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260928 20:24:48.419898 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:48.420241 30747 catalog_manager.cc:1347] Generated new cluster ID: dd56cbdb270f4c80b81049019f79dfba
I20260928 20:24:48.420295 30747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:48.435591 30747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:48.436350 30747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:48.439706 30747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5f95e7b25242f189d981c836dedc4b: Generated new TSK 0
I20260928 20:24:48.439960 30747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:48.449891 30740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.445601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.24.186.254:33361)
0928 20:24:48.445652 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.445654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.445700 (+ 46us) server_negotiation.cc:407] Connection header received
0928 20:24:48.445784 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.445785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.445800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.445869 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:24:48.446007 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.447085 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.447923 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.448254 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.448330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.448525 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:24:48.449517 (+ 992us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 20:24:48.449802 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:24:48.449957 30740 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57108: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 20:24:48.451586 30765 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:48.445488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33361 (local address 127.0.0.1:57108)
0928 20:24:48.445639 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:48.445657 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:48.445715 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:48.445885 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:48.445886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:48.445888 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:24:48.445938 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.445940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.447207 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.447208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.447852 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.447855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.448359 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.448360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.448409 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.448414 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:24:48.449635 (+ 1221us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 20:24:48.451484 (+ 1849us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.186.254:33361: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39903/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1660,"thread_start_us":85,"threads_started":1}
I20260928 20:24:48.451900 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30669
E20260928 20:24:48.463888 30766 webserver.cc:533] Webserver: sslize failed:
E20260928 20:24:48.463971 30766 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T20:24:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T20:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:48Z Disabled control of system clock
I20260928 20:24:48.497447 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38365
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38365
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/testchainca.cert with env {}
W20260928 20:24:48.577823 30770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:48.578110 30770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:48.578153 30770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:48.579330 30770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:48.579401 30770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:48.579423 30770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:48.579450 30770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:48.579483 30770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 20:24:48.579514 30770 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 20:24:48.579543 30770 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 20:24:48.581048 30770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:48.581333 30770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:48.581555 30770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:48.586366 30778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:48.586443 30776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:48.586376 30775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:48.586524 30770 server_base.cc:1034] running on GCE node
I20260928 20:24:48.586841 30770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:48.587119 30770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:48.588299 30770 hybrid_clock.cc:648] HybridClock initialized: now 1790627088588296 us; error 49 us; skew 500 ppm
I20260928 20:24:48.589614 30770 webserver.cc:460] Webserver started at http://127.24.186.254:42943/ using document root <none> and password file <none>
I20260928 20:24:48.589854 30770 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:48.589924 30770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:48.590078 30770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:48.591161 30770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "9288ca2e78dd4a978ae437a89e7b3c58"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "8e5e305661c8eb9a653034e6103538a4"
server_key_iv: "8543ae55c7b4de96d5a43a84e47d85de"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.591539 30770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "9288ca2e78dd4a978ae437a89e7b3c58"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "8e5e305661c8eb9a653034e6103538a4"
server_key_iv: "8543ae55c7b4de96d5a43a84e47d85de"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.592983 30770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:48.593809 30784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:48.594038 30770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:48.594143 30770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9288ca2e78dd4a978ae437a89e7b3c58"
format_stamp: "Formatted at 2026-09-28 20:24:48 on dist-test-slave-k477"
server_key: "8e5e305661c8eb9a653034e6103538a4"
server_key_iv: "8543ae55c7b4de96d5a43a84e47d85de"
server_key_version: "encryptionkey@0"
I20260928 20:24:48.594231 30770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:48.615164 30770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:48.615535 30770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:48.615694 30770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:48.619441 30770 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38365
I20260928 20:24:48.619499 30836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38365 every 8 connection(s)
I20260928 20:24:48.620012 30770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:48.621227 30837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:48.621836 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30770
I20260928 20:24:48.621955 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:48.622143 25323 external_mini_cluster.cc:1428] Setting key a4741a7c4be2c1b04f1a1ecc3a1f128e
I20260928 20:24:48.624300 30837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58: Bootstrap starting.
I20260928 20:24:48.624984 30837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:48.625298 30837 log.cc:824] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:48.626108 30837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58: No bootstrap required, opened a new log
I20260928 20:24:48.627961 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.622670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.24.186.254:38365)
0928 20:24:48.622858 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.622865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.622890 (+ 25us) server_negotiation.cc:407] Connection header received
0928 20:24:48.623294 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.623300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.623357 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.623470 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:48.623712 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.624675 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.625912 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.626403 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.626554 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.627251 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:48.627255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:48.627255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:24:48.627413 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:48.627444 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:24:48.627455 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:48.627503 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:48.627614 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 20:24:48.627706 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260928 20:24:48.628714 30837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } }
I20260928 20:24:48.628846 30837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:48.628909 30837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9288ca2e78dd4a978ae437a89e7b3c58, State: Initialized, Role: FOLLOWER
I20260928 20:24:48.628944 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:24:48.629056 30837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } }
I20260928 20:24:48.629136 30837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:48.629163 30837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:48.629207 30837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:48.629938 30837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } }
I20260928 20:24:48.630069 30837 leader_election.cc:302] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9288ca2e78dd4a978ae437a89e7b3c58; no voters:
I20260928 20:24:48.630245 30837 leader_election.cc:288] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:48.630303 30842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:48.630455 30842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [term 1 LEADER]: Becoming Leader. State: Replica: 9288ca2e78dd4a978ae437a89e7b3c58, State: Running, Role: LEADER
I20260928 20:24:48.630499 30837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:48.630545 30842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } }
I20260928 20:24:48.630928 30843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } } }
I20260928 20:24:48.630942 30844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9288ca2e78dd4a978ae437a89e7b3c58. Latest consensus state: current_term: 1 leader_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9288ca2e78dd4a978ae437a89e7b3c58" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38365 } } }
I20260928 20:24:48.631044 30843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:48.631053 30844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:48.631402 30851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:48.631915 30851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:48.633316 30851 catalog_manager.cc:1347] Generated new cluster ID: c9fd7fe111c24ab789212695999d3e54
I20260928 20:24:48.633371 30851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:48.644172 30851 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:48.644878 30851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:48.648326 30851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9288ca2e78dd4a978ae437a89e7b3c58: Generated new TSK 0
I20260928 20:24:48.648500 30851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:48.756559 30840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:48.663411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.24.186.254:38365)
0928 20:24:48.663477 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:48.663479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:48.663582 (+ 103us) server_negotiation.cc:407] Connection header received
0928 20:24:48.663679 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:48.663679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:48.663689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:48.663749 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0928 20:24:48.663901 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.664456 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.665578 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.665793 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.665846 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.666034 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 20:24:48.756334 (+ 90300us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 20:24:48.756497 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54492: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 20:24:48.756659 30840 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54492: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 20:24:48.756734 30866 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:48.663335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38365 (local address 127.0.0.1:54492)
0928 20:24:48.663518 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:48.663529 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:48.663602 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:48.663750 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:48.663751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:48.663753 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 20:24:48.663839 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.663842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.664543 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.664544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.665491 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:48.665495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:48.665874 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:48.665875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:48.665936 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:48.665942 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 20:24:48.756482 (+ 90540us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 20:24:48.756633 (+ 151us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.24.186.254:38365: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260928 20:24:48.756954 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30770
2026-09-28T20:24:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (285 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (96 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:49 dist-test-slave-k477 krb5kdc[30878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:49 dist-test-slave-k477 krb5kdc[30878](info): set up 2 sockets
Sep 28 20:24:49 dist-test-slave-k477 krb5kdc[30878](info): commencing operation
krb5kdc: starting...
W20260928 20:24:49.996661 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:24:50.127079 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38195
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:50.209859 30891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:50.210104 30891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:50.210140 30891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:50.211319 30891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:50.211382 30891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:50.211407 30891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:50.211436 30891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:50.211467 30891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:50.213001 30891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:50.213461 30891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:50.213716 30891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:50.218813 30897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.218834 30899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.218819 30896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.219431 30891 server_base.cc:1034] running on GCE node
I20260928 20:24:50.219693 30891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:50.220013 30891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:50.221179 30891 hybrid_clock.cc:648] HybridClock initialized: now 1790627090221171 us; error 38 us; skew 500 ppm
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.223721 30891 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:24:50.224303 30891 webserver.cc:460] Webserver started at http://127.24.186.254:44021/ using document root <none> and password file <none>
I20260928 20:24:50.224555 30891 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:50.224627 30891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:50.224781 30891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:50.225920 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "d6d648c08e074002a32d3437e3f03ffa"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "6ba22fb19f25c9fe18455fd11dac4718"
server_key_iv: "6a5fe8c844416d002862806d55c1457a"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.226307 30891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "d6d648c08e074002a32d3437e3f03ffa"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "6ba22fb19f25c9fe18455fd11dac4718"
server_key_iv: "6a5fe8c844416d002862806d55c1457a"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.227741 30891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:24:50.228945 30906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.229223 30891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:50.229339 30891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "d6d648c08e074002a32d3437e3f03ffa"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "6ba22fb19f25c9fe18455fd11dac4718"
server_key_iv: "6a5fe8c844416d002862806d55c1457a"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.229417 30891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:50.242998 30891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:50.245348 30891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:50.245534 30891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:50.249637 30891 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38195
I20260928 20:24:50.249684 30958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38195 every 8 connection(s)
I20260928 20:24:50.250044 30891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:50.250936 30959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:50.252410 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30891
I20260928 20:24:50.252511 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:50.252720 25323 external_mini_cluster.cc:1428] Setting key 4188059bb50fe3d4326f75fb37866d32
I20260928 20:24:50.253651 30959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa: Bootstrap starting.
I20260928 20:24:50.254350 30959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:50.254685 30959 log.cc:824] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:50.255474 30959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa: No bootstrap required, opened a new log
I20260928 20:24:50.257746 30959 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } }
I20260928 20:24:50.257942 30959 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:50.257977 30959 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d648c08e074002a32d3437e3f03ffa, State: Initialized, Role: FOLLOWER
I20260928 20:24:50.258126 30959 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } }
I20260928 20:24:50.258186 30959 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:50.258208 30959 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:50.258302 30959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:50.258863 30959 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } }
I20260928 20:24:50.258983 30959 leader_election.cc:302] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d648c08e074002a32d3437e3f03ffa; no voters:
I20260928 20:24:50.259176 30959 leader_election.cc:288] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:50.259197 30964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:50.259322 30964 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [term 1 LEADER]: Becoming Leader. State: Replica: d6d648c08e074002a32d3437e3f03ffa, State: Running, Role: LEADER
I20260928 20:24:50.259446 30959 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:50.259460 30964 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } }
I20260928 20:24:50.259842 30964 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d648c08e074002a32d3437e3f03ffa. Latest consensus state: current_term: 1 leader_uuid: "d6d648c08e074002a32d3437e3f03ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } } }
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:50.259855 30965 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d648c08e074002a32d3437e3f03ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d648c08e074002a32d3437e3f03ffa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38195 } } }
I20260928 20:24:50.260003 30964 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:50.260013 30965 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:50.260803 30972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:50.261348 30972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:50.263516 30972 catalog_manager.cc:1347] Generated new cluster ID: 5c579703647b416ca05d59d7638179f1
I20260928 20:24:50.263579 30972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:50.264577 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.253295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55756 (local address 127.24.186.254:38195)
0928 20:24:50.253458 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.253465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.253491 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:24:50.253931 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.253938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.253984 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.254116 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.255000 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.256130 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.257183 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.257589 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.257677 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.260606 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.260611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.260611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.260625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.262810 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.263266 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.263269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.263270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.263372 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.263610 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.263612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.263612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.263945 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.264094 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.264223 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.264367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260928 20:24:50.277354 30972 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:50.278120 30972 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:50.284343 30972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d648c08e074002a32d3437e3f03ffa: Generated new TSK 0
I20260928 20:24:50.284570 30972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:24:50.322255 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:50.427208 30985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:50.427457 30985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:50.427497 30985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:50.428777 30985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:50.428822 30985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:50.428850 30985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:50.430254 30985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:50.430436 30985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:50.430617 30985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:50.436090 30990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.436116 30985 server_base.cc:1034] running on GCE node
W20260928 20:24:50.436090 30993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.436120 30991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.436532 30985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:50.436807 30985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:50.437958 30985 hybrid_clock.cc:648] HybridClock initialized: now 1790627090437940 us; error 30 us; skew 500 ppm
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.441059 30985 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:24:50.441529 30985 webserver.cc:460] Webserver started at http://127.24.186.193:38105/ using document root <none> and password file <none>
I20260928 20:24:50.441758 30985 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:50.441804 30985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:50.441933 30985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:50.443008 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "95ef8419f0a9499a932040f60f92471e"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "37ab936dc9efd9572d5c97b437a57af1"
server_key_iv: "e81ac008ea453e3b34364849bc84cd31"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.443380 30985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "95ef8419f0a9499a932040f60f92471e"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "37ab936dc9efd9572d5c97b437a57af1"
server_key_iv: "e81ac008ea453e3b34364849bc84cd31"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.444852 30985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:50.445670 31000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.445883 30985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:50.445986 30985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "95ef8419f0a9499a932040f60f92471e"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "37ab936dc9efd9572d5c97b437a57af1"
server_key_iv: "e81ac008ea453e3b34364849bc84cd31"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.446069 30985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:50.468813 30985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:50.470827 30985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:50.470994 30985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:50.471241 30985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:50.471549 30985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:50.471601 30985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.471642 30985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:50.471666 30985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.477890 30985 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35771
I20260928 20:24:50.477937 31113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35771 every 8 connection(s)
I20260928 20:24:50.478303 30985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:50.487715 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.478984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:54713 (local address 127.24.186.254:38195)
0928 20:24:50.479034 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.479036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.479205 (+ 169us) server_negotiation.cc:407] Connection header received
0928 20:24:50.480245 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.480247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.480258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.480324 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.481027 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.481540 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.482548 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.482769 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.482827 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.485124 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.485126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.485126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.485133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.486520 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.486926 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.486927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.486927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.486981 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.487193 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.487194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.487194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.487301 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.487351 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.487578 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.487646 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:24:50.487920 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.478932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38195 (local address 127.24.186.193:54713)
0928 20:24:50.479091 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:50.479132 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:50.480128 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:50.480455 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:50.480459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:50.480732 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:50.480932 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.480936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.481648 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.481649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.482464 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.482466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.482864 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.482865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.482974 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.483498 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:50.483507 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:50.485003 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:50.486599 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.486605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.486607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.486855 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.487043 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.487043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.487044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.487139 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.487355 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:50.487360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:50.487472 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 20:24:50.487555 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:24:50.487674 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260928 20:24:50.488541 31114 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38195
I20260928 20:24:50.488544 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 30985
I20260928 20:24:50.488648 31114 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:50.488668 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:50.488845 31114 heartbeater.cc:511] Master 127.24.186.254:38195 requested a full tablet report, sending...
I20260928 20:24:50.488858 25323 external_mini_cluster.cc:1428] Setting key 1d81b947e3c5f37d0776bd9e1d8f50db
I20260928 20:24:50.489447 30922 ts_manager.cc:194] Registered new tserver with Master: 95ef8419f0a9499a932040f60f92471e (127.24.186.193:35771)
I20260928 20:24:50.490072 30922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:54713
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:24:50.515072 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:50.601863 31119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:50.602144 31119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:50.602188 31119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:50.603415 31119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:50.603489 31119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:50.603536 31119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:50.605178 31119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:50.605480 31119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:50.605728 31119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:50.610873 31125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.610875 31127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.610848 31124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.611200 31119 server_base.cc:1034] running on GCE node
I20260928 20:24:50.611408 31119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:50.611609 31119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:50.612890 31119 hybrid_clock.cc:648] HybridClock initialized: now 1790627090612844 us; error 31 us; skew 500 ppm
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.616281 31119 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:24:50.616797 31119 webserver.cc:460] Webserver started at http://127.24.186.194:44975/ using document root <none> and password file <none>
I20260928 20:24:50.617061 31119 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:50.617141 31119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:50.617295 31119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:50.618384 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "dfc85d9402cb4baf8f98843b7adea971"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "7b4de85b70abfba8f9c80757e481fae4"
server_key_iv: "0a46a435dd37b89eb1d421a253743d19"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.618753 31119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "dfc85d9402cb4baf8f98843b7adea971"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "7b4de85b70abfba8f9c80757e481fae4"
server_key_iv: "0a46a435dd37b89eb1d421a253743d19"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.620194 31119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:50.621022 31134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.621207 31119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:50.621310 31119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "dfc85d9402cb4baf8f98843b7adea971"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "7b4de85b70abfba8f9c80757e481fae4"
server_key_iv: "0a46a435dd37b89eb1d421a253743d19"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.621388 31119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:50.634147 31119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:50.636538 31119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:50.636715 31119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:50.636952 31119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:50.637267 31119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:50.637321 31119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.637364 31119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:50.637392 31119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.643139 31119 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:44119
I20260928 20:24:50.643216 31247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:44119 every 8 connection(s)
I20260928 20:24:50.643559 31119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:50.650774 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31119
I20260928 20:24:50.650906 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:50.651118 25323 external_mini_cluster.cc:1428] Setting key 5167c2715a81d182d3e22d7dceabd0ce
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:50.656088 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.644468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:56507 (local address 127.24.186.254:38195)
0928 20:24:50.644584 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.644586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.644754 (+ 168us) server_negotiation.cc:407] Connection header received
0928 20:24:50.645994 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.645996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.646017 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.646070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.647046 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.647620 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.648869 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.649155 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.649236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.652467 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.652469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.652469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.652479 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.654220 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.654961 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.654962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.654962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.655032 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.655345 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.655346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.655347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.655486 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.655577 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.655895 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.655987 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 20:24:50.656322 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.644384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38195 (local address 127.24.186.194:56507)
0928 20:24:50.644623 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:50.644672 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:50.645763 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:50.646172 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:50.646183 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:50.646630 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:50.646921 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.646927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.647736 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.647738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.648738 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.648741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.649257 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.649258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.649545 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.650268 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:50.650278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:50.652360 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:50.654413 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.654422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.654426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.654779 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.655126 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.655127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.655127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.655262 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.655559 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:50.655567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:50.655721 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 20:24:50.655869 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 20:24:50.656067 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":89,"threads_started":1}
I20260928 20:24:50.656793 31248 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38195
I20260928 20:24:50.656912 31248 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:50.657109 31248 heartbeater.cc:511] Master 127.24.186.254:38195 requested a full tablet report, sending...
I20260928 20:24:50.657507 30922 ts_manager.cc:194] Registered new tserver with Master: dfc85d9402cb4baf8f98843b7adea971 (127.24.186.194:44119)
I20260928 20:24:50.657902 30922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:56507
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:24:50.679301 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:50.764748 31253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:50.764964 31253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:50.764995 31253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:50.766141 31253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:50.766177 31253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:50.766206 31253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:50.767694 31253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:50.767971 31253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:50.768229 31253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:50.773545 31258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:50.773924 31259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.773988 31253 server_base.cc:1034] running on GCE node
W20260928 20:24:50.774137 31261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:50.774398 31253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:50.774680 31253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:50.775887 31253 hybrid_clock.cc:648] HybridClock initialized: now 1790627090775864 us; error 41 us; skew 500 ppm
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.778515 31253 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:24:50.779109 31253 webserver.cc:460] Webserver started at http://127.24.186.195:35557/ using document root <none> and password file <none>
I20260928 20:24:50.779371 31253 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:50.779443 31253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:50.779590 31253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:50.780709 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "1216fa015f1a426a88db60140dfe4dbf"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "4c5013f26158fb7ccfa18c9215513221"
server_key_iv: "b9379e9d6f0a6e678a16657ec09f36f4"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.781068 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "1216fa015f1a426a88db60140dfe4dbf"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "4c5013f26158fb7ccfa18c9215513221"
server_key_iv: "b9379e9d6f0a6e678a16657ec09f36f4"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.782503 31253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:50.783308 31268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.783481 31253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:50.783573 31253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "1216fa015f1a426a88db60140dfe4dbf"
format_stamp: "Formatted at 2026-09-28 20:24:50 on dist-test-slave-k477"
server_key: "4c5013f26158fb7ccfa18c9215513221"
server_key_iv: "b9379e9d6f0a6e678a16657ec09f36f4"
server_key_version: "encryptionkey@0"
I20260928 20:24:50.783658 31253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:50.796420 31253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:50.798374 31253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:50.798554 31253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:50.798800 31253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:50.799106 31253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:50.799160 31253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.799201 31253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:50.799224 31253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:50.805194 31253 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:37147
I20260928 20:24:50.805238 31381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:37147 every 8 connection(s)
I20260928 20:24:50.805563 31253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:24:50.814651 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31253
I20260928 20:24:50.814765 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:50.814952 25323 external_mini_cluster.cc:1428] Setting key 667a39d84b72d156e58ba6b83f7b180b
I20260928 20:24:50.815328 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.806304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:55787 (local address 127.24.186.254:38195)
0928 20:24:50.806384 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.806387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.806579 (+ 192us) server_negotiation.cc:407] Connection header received
0928 20:24:50.807572 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.807573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.807584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.807639 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.808363 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.808871 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.809834 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.810024 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.810089 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.812281 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.812282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.812283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.812292 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.813723 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.814223 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.814224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.814225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.814291 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.814548 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.814549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.814549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.814680 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.814763 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.815127 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.815215 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 20:24:50.815596 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.806254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38195 (local address 127.24.186.195:55787)
0928 20:24:50.806427 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:50.806464 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:50.807460 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:50.807710 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:50.807714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:50.808069 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:50.808272 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.808276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.808948 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.808949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.809733 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:50.809735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.810101 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.810102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:50.810202 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.810702 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:50.810710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:50.812176 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:50.813854 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.813862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.813865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.814105 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.814359 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:50.814359 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:50.814360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:50.814478 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:50.814785 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:50.814792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:50.814970 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 20:24:50.815105 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 20:24:50.815318 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260928 20:24:50.816063 31382 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38195
I20260928 20:24:50.816170 31382 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:50.816365 31382 heartbeater.cc:511] Master 127.24.186.254:38195 requested a full tablet report, sending...
I20260928 20:24:50.816753 30923 ts_manager.cc:194] Registered new tserver with Master: 1216fa015f1a426a88db60140dfe4dbf (127.24.186.195:37147)
I20260928 20:24:50.817097 30923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:55787
I20260928 20:24:50.827025 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/test-admin.keytab.
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.844326 25323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 20:24:50.846107 25323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc' of type FILE
W20260928 20:24:50.846127 31389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:50.845276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.24.186.193:35771)
0928 20:24:50.845447 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.845452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.845471 (+ 19us) server_negotiation.cc:407] Connection header received
0928 20:24:50.845499 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.845501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.845515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.845582 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.845989 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35416: BlockingRecv error: recv got EOF from 127.0.0.1:35416 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.847249 25323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.848299 25323 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:50.856873 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.849015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.24.186.193:35771)
0928 20:24:50.849088 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.849090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.849106 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:24:50.849131 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.849132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.849143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.849195 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.849651 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.850398 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.851150 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.851493 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.851559 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.854159 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.854169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.854170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.854178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.855880 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.856191 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.856194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.856194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.856248 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.856410 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.856410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.856410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.856548 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.856653 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.856728 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.856789 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 20:24:50.858695 31389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:50.858184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.24.186.193:35771)
0928 20:24:50.858217 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.858219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.858275 (+ 56us) server_negotiation.cc:407] Connection header received
0928 20:24:50.858325 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.858325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.858334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.858381 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.858643 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35422: BlockingRecv error: recv got EOF from 127.0.0.1:35422 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 20:24:50.858804 25323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc' of type FILE
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.859925 25323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.860868 25323 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:50.867660 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.861384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.24.186.193:35771)
0928 20:24:50.861429 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.861431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.861453 (+ 22us) server_negotiation.cc:407] Connection header received
0928 20:24:50.861513 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.861514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.861523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.861574 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.861936 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.862453 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.863096 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.863300 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.863351 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.865353 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.865355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.865355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.865361 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.866759 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.867050 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.867051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.867051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.867099 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.867287 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.867288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.867288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.867378 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.867444 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.867533 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.867592 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 20:24:50.869442 31389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:50.868850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.24.186.193:35771)
0928 20:24:50.868889 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.868891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.868984 (+ 93us) server_negotiation.cc:407] Connection header received
0928 20:24:50.869050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.869050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.869061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.869109 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.869388 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35446: BlockingRecv error: recv got EOF from 127.0.0.1:35446 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 20:24:50.869554 25323 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790627060930553-25323-0/krb5kdc/krb5cc' of type FILE
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.870581 25323 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:50.871441 25323 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[30878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:24:50.878482 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:50.871985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.24.186.193:35771)
0928 20:24:50.872023 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:50.872025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:50.872124 (+ 99us) server_negotiation.cc:407] Connection header received
0928 20:24:50.872168 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:50.872168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:50.872191 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:50.872231 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:50.872567 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.873042 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.873726 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:50.874090 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:50.874157 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:50.876327 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:50.876328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:50.876329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:50.876335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:50.877554 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.877891 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.877892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.877892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.877948 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:50.878113 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:50.878113 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:50.878113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:50.878208 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:50.878282 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:50.878360 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:24:50.878419 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:50.879225 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30985
I20260928 20:24:50.884444 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31119
I20260928 20:24:50.888960 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31253
I20260928 20:24:50.893237 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 30891
2026-09-28T20:24:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1946 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[31402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[31402](info): set up 2 sockets
Sep 28 20:24:50 dist-test-slave-k477 krb5kdc[31402](info): commencing operation
krb5kdc: starting...
W20260928 20:24:51.932646 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab.
Entry for principal oryx/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab.
I20260928 20:24:52.054971 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33525
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab
--principal=oryx/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:52.141799 31415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:52.142035 31415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:52.142079 31415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:52.143460 31415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:52.143517 31415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:52.143532 31415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:52.143544 31415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:52.143560 31415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:52.145242 31415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab
--principal=oryx/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:52.145529 31415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:52.145783 31415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:52.150854 31420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.150910 31423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.150910 31421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.151006 31415 server_base.cc:1034] running on GCE node
I20260928 20:24:52.151346 31415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:52.151669 31415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:52.152911 31415 hybrid_clock.cc:648] HybridClock initialized: now 1790627092152883 us; error 43 us; skew 500 ppm
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:52.155581 31415 init.cc:377] Logged in from keytab as oryx/127.24.186.254@KRBTEST.COM (short username oryx)
I20260928 20:24:52.156113 31415 webserver.cc:460] Webserver started at http://127.24.186.254:43193/ using document root <none> and password file <none>
I20260928 20:24:52.156350 31415 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.156422 31415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.156576 31415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:52.157655 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "37cf5c1738184d959f42e8f4d3ac8ec4"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "5abced690f6acb727cc3d72f226d47a5"
server_key_iv: "f2d3de76ffb98ac9418d6bd7d4c8a0d0"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.158039 31415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "37cf5c1738184d959f42e8f4d3ac8ec4"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "5abced690f6acb727cc3d72f226d47a5"
server_key_iv: "f2d3de76ffb98ac9418d6bd7d4c8a0d0"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.159474 31415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:52.160460 31430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.160645 31415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:24:52.160759 31415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "37cf5c1738184d959f42e8f4d3ac8ec4"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "5abced690f6acb727cc3d72f226d47a5"
server_key_iv: "f2d3de76ffb98ac9418d6bd7d4c8a0d0"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.160838 31415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.179953 31415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:52.182260 31415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:52.182461 31415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:52.187505 31415 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:33525
I20260928 20:24:52.188171 31415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:52.188117 31482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:33525 every 8 connection(s)
I20260928 20:24:52.189391 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:52.190501 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31415
I20260928 20:24:52.190598 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:52.190820 25323 external_mini_cluster.cc:1428] Setting key 7096c7432540e15856e9fd0508476d8f
I20260928 20:24:52.192778 31483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Bootstrap starting.
I20260928 20:24:52.193586 31483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:52.193951 31483 log.cc:824] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:52.194770 31483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: No bootstrap required, opened a new log
I20260928 20:24:52.197443 31483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } }
I20260928 20:24:52.197602 31483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:52.197633 31483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37cf5c1738184d959f42e8f4d3ac8ec4, State: Initialized, Role: FOLLOWER
I20260928 20:24:52.197788 31483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } }
I20260928 20:24:52.197902 31483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:52.197968 31483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:52.198021 31483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:52.198839 31483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } }
I20260928 20:24:52.198974 31483 leader_election.cc:302] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37cf5c1738184d959f42e8f4d3ac8ec4; no voters:
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:52.199165 31483 leader_election.cc:288] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:52.199226 31488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:52.199398 31488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 37cf5c1738184d959f42e8f4d3ac8ec4, State: Running, Role: LEADER
I20260928 20:24:52.199467 31483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:52.199501 31488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } }
I20260928 20:24:52.199841 31488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cf5c1738184d959f42e8f4d3ac8ec4. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } }
I20260928 20:24:52.199921 31488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:52.200047 31490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } }
I20260928 20:24:52.200142 31493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:52.200168 31490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:52.200695 31493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:52.202737 31493 catalog_manager.cc:1347] Generated new cluster ID: 5280696460894460b2ce82cc9a75e73e
I20260928 20:24:52.202788 31493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:52.204491 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.191420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.24.186.254:33525)
0928 20:24:52.191645 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.191657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.191695 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:24:52.192417 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.192428 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.192500 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.192680 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.193389 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.194956 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.195768 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.196540 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.196684 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.199777 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.199782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.199782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.199821 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.202325 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.202999 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.203003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.203004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.203108 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.203349 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.203350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.203350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.203652 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.203776 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.204160 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.204301 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260928 20:24:52.212563 31493 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:52.213078 31493 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:52.224804 31493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Generated new TSK 0
I20260928 20:24:52.225028 31493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
I20260928 20:24:52.258205 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:52.344178 31509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:52.344405 31509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:52.344431 31509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:52.345564 31509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:52.345602 31509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:52.345628 31509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:52.347182 31509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:52.347473 31509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:52.347759 31509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:52.352844 31514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.352844 31517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.353101 31509 server_base.cc:1034] running on GCE node
W20260928 20:24:52.353106 31515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.353336 31509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:52.353595 31509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:52.354743 31509 hybrid_clock.cc:648] HybridClock initialized: now 1790627092354719 us; error 35 us; skew 500 ppm
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:52.357451 31509 init.cc:377] Logged in from keytab as oryx/127.24.186.193@KRBTEST.COM (short username oryx)
I20260928 20:24:52.357932 31509 webserver.cc:460] Webserver started at http://127.24.186.193:39291/ using document root <none> and password file <none>
I20260928 20:24:52.358183 31509 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.358229 31509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.358397 31509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:52.359503 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "c456a93ad7bf4c76bdc5c6045231d875"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "7c87200daf594fd911009695565940f0"
server_key_iv: "bdd9ed21b5c2fe32b097dcff85180a08"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.359925 31509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "c456a93ad7bf4c76bdc5c6045231d875"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "7c87200daf594fd911009695565940f0"
server_key_iv: "bdd9ed21b5c2fe32b097dcff85180a08"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.361320 31509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.362076 31524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.362310 31509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.362401 31509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "c456a93ad7bf4c76bdc5c6045231d875"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "7c87200daf594fd911009695565940f0"
server_key_iv: "bdd9ed21b5c2fe32b097dcff85180a08"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.362475 31509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.371368 31509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:52.373677 31509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:52.373855 31509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:52.374154 31509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:52.374514 31509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:52.374575 31509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.374614 31509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:52.374650 31509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.381085 31509 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:38787
I20260928 20:24:52.381125 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:38787 every 8 connection(s)
I20260928 20:24:52.381465 31509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:52.383473 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31509
I20260928 20:24:52.383574 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:52.383787 25323 external_mini_cluster.cc:1428] Setting key 56ad0a27857365f33b2abcbf7c736ada
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:24:52.392603 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.382432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:54199 (local address 127.24.186.254:33525)
0928 20:24:52.382509 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.382512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.382721 (+ 209us) server_negotiation.cc:407] Connection header received
0928 20:24:52.383203 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.383205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.383215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.383316 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.384541 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.385300 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.386419 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.386765 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.386846 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.389453 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.389457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.389458 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.389470 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.390981 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.391532 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.391534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.391534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.391600 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.391894 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.391895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.391895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.392023 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.392105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.392399 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.392501 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:24:52.392832 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.382389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:54199)
0928 20:24:52.382568 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.382607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.383092 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.383383 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.383386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.384126 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.384331 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.384337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.385414 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.385415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.386328 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.386331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.386866 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.386867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.386978 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.387555 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.387563 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.389218 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.391177 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.391185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.391188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.391438 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.391680 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.391681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.391681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.391795 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.392094 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.392101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.392258 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.392367 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.392544 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260928 20:24:52.393396 31638 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:52.393509 31638 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:52.393710 31638 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:52.394333 31447 ts_manager.cc:194] Registered new tserver with Master: c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:52.394975 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:54199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
I20260928 20:24:52.411048 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:52.507928 31643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:52.508163 31643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:52.508189 31643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:52.509509 31643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:52.509559 31643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:52.509589 31643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:52.511163 31643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:52.511447 31643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:52.511724 31643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:52.517391 31651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.517421 31648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.517675 31643 server_base.cc:1034] running on GCE node
W20260928 20:24:52.517444 31649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.518085 31643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:52.518363 31643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:52.519556 31643 hybrid_clock.cc:648] HybridClock initialized: now 1790627092519527 us; error 37 us; skew 500 ppm
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:52.522234 31643 init.cc:377] Logged in from keytab as oryx/127.24.186.194@KRBTEST.COM (short username oryx)
I20260928 20:24:52.522761 31643 webserver.cc:460] Webserver started at http://127.24.186.194:45109/ using document root <none> and password file <none>
I20260928 20:24:52.523013 31643 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.523083 31643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.523236 31643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:52.524430 31643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "7f43d62f60d6443bb7e4a47d028df299"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "6d1c3e8f8a9327e18fe8b6e100c9e3c1"
server_key_iv: "8b758f772dfb6503ce5ea05ac36ca7ac"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.524814 31643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "7f43d62f60d6443bb7e4a47d028df299"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "6d1c3e8f8a9327e18fe8b6e100c9e3c1"
server_key_iv: "8b758f772dfb6503ce5ea05ac36ca7ac"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.526298 31643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:52.527338 31658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.527542 31643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.527665 31643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "7f43d62f60d6443bb7e4a47d028df299"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "6d1c3e8f8a9327e18fe8b6e100c9e3c1"
server_key_iv: "8b758f772dfb6503ce5ea05ac36ca7ac"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.527747 31643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.543612 31643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:52.545697 31643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:52.545872 31643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:52.546119 31643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:52.546445 31643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:52.546505 31643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.546542 31643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:52.546591 31643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.552723 31643 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:42867
I20260928 20:24:52.552775 31771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:42867 every 8 connection(s)
I20260928 20:24:52.553153 31643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:52.557211 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31643
I20260928 20:24:52.557309 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:52.557525 25323 external_mini_cluster.cc:1428] Setting key 473614a5a0b90dcba5c29ccb2ae3c9eb
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:52.564067 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.554010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:41311 (local address 127.24.186.254:33525)
0928 20:24:52.554135 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.554138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.554320 (+ 182us) server_negotiation.cc:407] Connection header received
0928 20:24:52.554894 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.554895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.554909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.554960 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.556003 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.556546 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.557726 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.557961 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.558030 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.560879 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.560881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.560882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.560889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.562337 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.562920 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.562922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.562922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.562990 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.563315 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.563317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.563317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.563443 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.563498 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.563890 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.563978 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 20:24:52.564374 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.553924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:41311)
0928 20:24:52.554184 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.554234 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.554773 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.555027 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.555031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.555630 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.555901 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.555906 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.556664 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.556665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.557619 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.557621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.558044 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.558045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.558230 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.558779 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.558791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.560762 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.562446 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.562454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.562457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.562790 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.563070 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.563071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.563071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.563227 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.563515 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.563522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.563708 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.563859 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.564062 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260928 20:24:52.564888 31772 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:52.565004 31772 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:52.565214 31772 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:52.565673 31447 ts_manager.cc:194] Registered new tserver with Master: 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
I20260928 20:24:52.566064 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:41311
WARNING: no policy specified for oryx/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
I20260928 20:24:52.586364 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:52.675410 31777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:52.675635 31777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:52.675659 31777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:52.676879 31777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:52.676926 31777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:52.676954 31777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:52.678422 31777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:52.678685 31777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:52.678927 31777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:52.684018 31782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.684005 31785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.684077 31777 server_base.cc:1034] running on GCE node
W20260928 20:24:52.684005 31783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.684383 31777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:52.684588 31777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:52.685745 31777 hybrid_clock.cc:648] HybridClock initialized: now 1790627092685721 us; error 42 us; skew 500 ppm
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:52.693643 31777 init.cc:377] Logged in from keytab as oryx/127.24.186.195@KRBTEST.COM (short username oryx)
I20260928 20:24:52.694192 31777 webserver.cc:460] Webserver started at http://127.24.186.195:38495/ using document root <none> and password file <none>
I20260928 20:24:52.694427 31777 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.694473 31777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.694646 31777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:52.695714 31777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "d76b21af060c44dc82dd9ce32f61981e"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "c90997576e0fb35249282f891037c799"
server_key_iv: "af110a0433147f1ce07538c87408ffbb"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.696146 31777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "d76b21af060c44dc82dd9ce32f61981e"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "c90997576e0fb35249282f891037c799"
server_key_iv: "af110a0433147f1ce07538c87408ffbb"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.697526 31777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.698529 31792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.698802 31777 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.698928 31777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "d76b21af060c44dc82dd9ce32f61981e"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "c90997576e0fb35249282f891037c799"
server_key_iv: "af110a0433147f1ce07538c87408ffbb"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.699018 31777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.730402 31777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:52.732702 31777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:52.732890 31777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:52.733148 31777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:52.733477 31777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:52.733532 31777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.733572 31777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:52.733597 31777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.739627 31777 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45335
I20260928 20:24:52.739677 31905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45335 every 8 connection(s)
I20260928 20:24:52.740099 31777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:52.742234 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31777
I20260928 20:24:52.742316 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:52.742522 25323 external_mini_cluster.cc:1428] Setting key e323bd7d44259978630205a33a1dedb3
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:52.749744 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.740807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:49441 (local address 127.24.186.254:33525)
0928 20:24:52.740870 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.740872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.741075 (+ 203us) server_negotiation.cc:407] Connection header received
0928 20:24:52.741630 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.741631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.741642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.741708 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.742838 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.743346 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.744447 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.744642 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.744721 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.746849 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.746851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.746851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.746874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.748296 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.748717 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.748718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.748718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.748780 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.749047 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.749048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.749048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.749186 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.749266 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.749560 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.749643 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:24:52.749948 31908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.740793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:49441)
0928 20:24:52.740954 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.740998 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.741504 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.741787 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.741791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.742470 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.742731 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.742736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.743457 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.743458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.744351 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.744354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.744729 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.744730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.744865 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.745360 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.745368 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.746740 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.748402 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.748409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.748411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.748638 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.748849 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.748850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.748850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.748972 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.749256 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.749264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.749427 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.749540 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.749714 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260928 20:24:52.750455 31906 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:52.750554 31906 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:52.750702 31906 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:52.751081 31447 ts_manager.cc:194] Registered new tserver with Master: d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335)
I20260928 20:24:52.751474 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:49441
I20260928 20:24:52.754598 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.186.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.253.keytab.
Entry for principal oryx/127.24.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.253.keytab.
I20260928 20:24:52.780860 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:38315
--webserver_interface=127.24.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.253.keytab
--principal=oryx/127.24.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33525,127.24.186.253:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:52.869784 31911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:52.870031 31911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:52.870069 31911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:52.871241 31911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:52.871315 31911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:52.871340 31911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:52.871372 31911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:52.871397 31911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:52.872995 31911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33525,127.24.186.253:38315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.253.keytab
--principal=oryx/127.24.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
W20260928 20:24:52.873137 31911 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.24.186.254:33525,127.24.186.253:38315'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 20:24:52.873309 31911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:52.873546 31911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:52.878429 31916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.878551 31911 server_base.cc:1034] running on GCE node
W20260928 20:24:52.878422 31917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:52.878422 31919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:52.878973 31911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:52.879308 31911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:52.880483 31911 hybrid_clock.cc:648] HybridClock initialized: now 1790627092880465 us; error 39 us; skew 500 ppm
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:52.883059 31911 init.cc:377] Logged in from keytab as oryx/127.24.186.253@KRBTEST.COM (short username oryx)
I20260928 20:24:52.883556 31911 webserver.cc:460] Webserver started at http://127.24.186.253:40689/ using document root <none> and password file <none>
I20260928 20:24:52.883841 31911 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.883906 31911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.884050 31911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:52.885139 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/instance:
uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "45035f963ba871d333ae4795e284755f"
server_key_iv: "be6a80a4b0ff32d0081861614b0f13b9"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.885509 31911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal/instance:
uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "45035f963ba871d333ae4795e284755f"
server_key_iv: "be6a80a4b0ff32d0081861614b0f13b9"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.886869 31911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:24:52.887645 31926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.887933 31911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:52.888021 31911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "45035f963ba871d333ae4795e284755f"
server_key_iv: "be6a80a4b0ff32d0081861614b0f13b9"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.888095 31911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.909729 31911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:52.912123 31911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:52.912246 31911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:52.916317 31911 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.253:38315
I20260928 20:24:52.916373 31978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.253:38315 every 8 connection(s)
I20260928 20:24:52.916708 31911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
I20260928 20:24:52.917654 31979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:52.919158 31979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.253@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:52.926208 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 31911
I20260928 20:24:52.929062 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.919967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.24.186.254:33525)
0928 20:24:52.920059 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.920061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.920236 (+ 175us) server_negotiation.cc:407] Connection header received
0928 20:24:52.920762 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.920763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.920778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.920829 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.921451 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.921947 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.922957 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.923152 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.923238 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.925523 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.925525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.925525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.925532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.927308 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.927897 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.927898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.927899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.927960 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.928294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.928296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.928296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.928507 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.928582 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.928873 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.928960 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:24:52.929430 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.919882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38384)
0928 20:24:52.920096 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.920149 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.920643 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.920900 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.920904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.921171 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.921357 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.921361 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.922046 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.922047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.922862 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.922864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.923246 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.923247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.923373 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.923950 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.923958 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.925432 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.927403 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.927410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.927413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.927761 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.928038 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.928039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.928040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.928196 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.928583 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.928588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.928722 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.928805 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.929200 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":206,"thread_start_us":105,"threads_started":1}
I20260928 20:24:52.930228 31979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.253@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
I20260928 20:24:52.936198 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.926787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.24.186.253:38315)
0928 20:24:52.927137 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.927143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.927168 (+ 25us) server_negotiation.cc:407] Connection header received
0928 20:24:52.927193 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.927196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.927210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.927293 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.928190 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.929240 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.930129 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.930592 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.930681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.933184 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.933191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.933192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.933198 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.934810 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.935223 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.935226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.935226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.935278 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.935505 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.935505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.935506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.935634 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.935751 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.935902 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.935984 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":88,"threads_started":1}
I20260928 20:24:52.937737 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.930484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:50964)
0928 20:24:52.930557 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.930567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.930610 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.930788 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.930789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.930943 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.930996 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.930998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.931658 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.931659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.932211 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.932213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.932569 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.932570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.932630 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.933032 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.933036 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.934358 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.936719 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.936720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.936721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.936910 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.937110 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.937111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.937112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.937171 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.937421 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.937422 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.937457 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.937519 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.937636 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":3}
I20260928 20:24:52.937846 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.930561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.24.186.253:38315)
0928 20:24:52.930692 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.930694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.930699 (+ 5us) server_negotiation.cc:407] Connection header received
0928 20:24:52.930714 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.930714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.930723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.930779 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.931070 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.931584 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.932287 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.932487 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.932569 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.934440 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.934441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.934442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.934448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.936634 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.936992 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.936993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.936993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.937047 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.937255 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.937256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.937257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.937356 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.937431 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.937537 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.937697 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":115,"thread_start_us":105,"threads_started":1}
I20260928 20:24:52.939033 31979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Bootstrap starting.
I20260928 20:24:52.939647 31979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:52.939989 31979 log.cc:824] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:52.940716 31979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: No bootstrap required, opened a new log
I20260928 20:24:52.942101 31979 raft_consensus.cc:348] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } }
I20260928 20:24:52.942226 31979 raft_consensus.cc:374] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:52.942250 31979 raft_consensus.cc:729] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87c6b879e7c4a8ba3e71527cd60dbce, State: Initialized, Role: FOLLOWER
I20260928 20:24:52.942379 31979 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } }
I20260928 20:24:52.942714 31979 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:52.942689 31986 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } } }
I20260928 20:24:52.942792 31986 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:24:52.944499 31997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:52.944561 31997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:52.946125 31911 master_runner.cc:428] Detected that this master f87c6b879e7c4a8ba3e71527cd60dbce is joining an existing cluster
I20260928 20:24:52.946182 31911 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.186.253:38315
I20260928 20:24:52.953186 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.947148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.24.186.254:33525)
0928 20:24:52.947200 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.947202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.947359 (+ 157us) server_negotiation.cc:407] Connection header received
0928 20:24:52.947437 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.947438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.947453 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.947511 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.947971 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.948473 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.949212 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.949405 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.949476 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.950489 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.950490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.950491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.950498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.951945 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.952351 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.952353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.952353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.952420 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.952631 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.952632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.952632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.952764 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.952839 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.953010 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.953084 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:24:52.953225 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.947111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38388)
0928 20:24:52.947282 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.947299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.947375 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.947515 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.947516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.947747 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.947881 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.947884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.948560 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.948561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.949117 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.949119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.949486 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.949487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.949554 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.950031 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.950036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.950379 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.952038 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.952039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.952040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.952266 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.952491 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.952492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.952492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.952558 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.952840 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.952841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.952897 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.952973 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.953084 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260928 20:24:52.959378 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.952051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.24.186.253:38315)
0928 20:24:52.952098 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.952100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.952350 (+ 250us) server_negotiation.cc:407] Connection header received
0928 20:24:52.952439 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.952440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.952449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.952497 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.953182 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.953911 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.954955 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.955337 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.955384 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.956378 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.956379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.956379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.956386 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.958186 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.958557 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.958557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.958558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.958605 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.958788 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.958788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.958789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.958924 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.958989 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.959163 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.959279 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":23}
I20260928 20:24:52.959712 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.952124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:50968)
0928 20:24:52.952294 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.952305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.952372 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.952532 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.952533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.953023 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.953078 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.953080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.954078 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.954078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.954854 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.954857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.955462 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.955463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.955514 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.955888 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.955892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.956285 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.958279 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.958280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.958281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.958483 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.958668 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.958668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.958668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.958716 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.958997 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.958998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.959045 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.959126 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.959611 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
Sep 28 20:24:52 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.254@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
I20260928 20:24:52.963964 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.956435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.24.186.253:38315)
0928 20:24:52.956501 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.956503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.956780 (+ 277us) server_negotiation.cc:407] Connection header received
0928 20:24:52.956884 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.956885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.956895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.956951 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.957487 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.957950 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.958765 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.959078 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.959152 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.961246 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.961247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.961248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.961255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.962588 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.963053 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.963054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.963055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.963111 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.963354 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.963355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.963355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.963504 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.963581 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.963752 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.963870 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:24:52.964012 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.956249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:50974)
0928 20:24:52.956686 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.956733 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.956805 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.956958 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.956961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.957334 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.957405 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.957408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.958037 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.958038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.958641 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.958643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.959218 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.959220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.959299 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.959754 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.959777 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.961157 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.962684 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.962692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.962694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.962978 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.963198 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.963199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.963199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.963262 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.963576 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.963582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.963630 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.963705 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.963863 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":88,"threads_started":1}
I20260928 20:24:52.964687 31447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.186.253:38315: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
member_type: NON_VOTER
last_known_addr {
host: "127.24.186.253"
port: 38315
}
attrs {
promote: true
}
}
dest_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4"
cas_config_opid_index: -1
I20260928 20:24:52.965054 31447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } }
I20260928 20:24:52.965590 32007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } }
I20260928 20:24:52.965876 31953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:52.966725 32007 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.186.253:38315
I20260928 20:24:52.966825 31953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 20:24:52.966854 32007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } }
I20260928 20:24:52.966908 32007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:52.967000 31911 master.cc:537] Master@127.24.186.253:38315 shutting down...
I20260928 20:24:52.967139 32007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 20:24:52.967832 31434 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 20:24:52.968205 31434 consensus_peers.cc:487] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 -> Peer f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253:38315): Couldn't send request to peer f87c6b879e7c4a8ba3e71527cd60dbce. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 20:24:52.969409 31911 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:52.969493 31911 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:52.969542 31911 tablet_replica.cc:330] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: stopping tablet replica
I20260928 20:24:52.982173 31911 master.cc:552] Master@127.24.186.253:38315 shutdown complete.
I20260928 20:24:52.982239 31911 master_runner.cc:305] Clearing existing system tablet
I20260928 20:24:52.982373 31911 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:52.982412 31911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:52.983151 31911 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260928 20:24:52.983693 32011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:52.983851 31911 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:52.983919 31911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:52.983959 31911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "45035f963ba871d333ae4795e284755f"
server_key_iv: "be6a80a4b0ff32d0081861614b0f13b9"
server_key_version: "encryptionkey@0"
I20260928 20:24:52.984591 31911 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:52.986274 31911 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 20:24:52.986332 31911 log.cc:1196] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 20:24:52.986532 31911 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Deleting consensus metadata
I20260928 20:24:52.986660 31911 master_runner.cc:315] Copying system tablet from 127.24.186.254:33525
I20260928 20:24:52.987301 31911 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: tablet copy: Beginning tablet copy session from remote peer at address 127.24.186.254:33525
I20260928 20:24:52.993799 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.987693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.24.186.254:33525)
0928 20:24:52.987740 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:52.987745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:52.988240 (+ 495us) server_negotiation.cc:407] Connection header received
0928 20:24:52.988337 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:52.988338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:52.988350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:52.988407 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:52.988732 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.989237 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.989942 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.990154 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.990240 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.991215 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:52.991217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:52.991218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:52.991224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:52.992648 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.993010 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.993011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.993011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.993071 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:52.993268 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:52.993268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:52.993269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:52.993382 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:52.993460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:52.993630 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:24:52.993704 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:24:52.993813 32017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:52.988024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38402)
0928 20:24:52.988165 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:52.988177 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:52.988256 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:52.988413 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:52.988414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:52.988583 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:52.988644 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.988646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.989323 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.989324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.989857 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:52.989858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:52.990238 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:52.990239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:52.990310 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:52.990822 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:52.990826 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:52.991120 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:52.992748 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.992749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.992750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.992933 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.993141 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:52.993142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:52.993142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:52.993205 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:52.993450 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:52.993451 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:52.993498 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:24:52.993587 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:24:52.993707 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260928 20:24:52.994098 31467 tablet_copy_service.cc:140] P 37cf5c1738184d959f42e8f4d3ac8ec4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f87c6b879e7c4a8ba3e71527cd60dbce ({username='oryx', principal='oryx/127.24.186.253@KRBTEST.COM'} at 127.0.0.1:38402)
I20260928 20:24:52.994204 31467 tablet_copy_service.cc:161] P 37cf5c1738184d959f42e8f4d3ac8ec4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f87c6b879e7c4a8ba3e71527cd60dbce at {username='oryx', principal='oryx/127.24.186.253@KRBTEST.COM'} at 127.0.0.1:38402: session id = f87c6b879e7c4a8ba3e71527cd60dbce-00000000000000000000000000000000
I20260928 20:24:52.994848 31467 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 20:24:52.995468 31911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 20:24:52.996505 31911 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: tablet copy: Starting download of 0 data blocks...
I20260928 20:24:52.996665 31911 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: tablet copy: Starting download of 1 WAL segments...
I20260928 20:24:52.997577 31911 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 20:24:52.998343 31467 tablet_copy_service.cc:342] P 37cf5c1738184d959f42e8f4d3ac8ec4: Request end of tablet copy session f87c6b879e7c4a8ba3e71527cd60dbce-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.186.253@KRBTEST.COM'} at 127.0.0.1:38402
I20260928 20:24:52.998404 31467 tablet_copy_service.cc:433] P 37cf5c1738184d959f42e8f4d3ac8ec4: ending tablet copy session f87c6b879e7c4a8ba3e71527cd60dbce-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f87c6b879e7c4a8ba3e71527cd60dbce
I20260928 20:24:53.000793 31911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.002032 32019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.002134 31911 server_base.cc:1034] running on GCE node
W20260928 20:24:53.002223 32021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.002046 32018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.002424 31911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.002583 31911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.003705 31911 hybrid_clock.cc:648] HybridClock initialized: now 1790627093003697 us; error 25 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.005020 31911 init.cc:377] Logged in from keytab as oryx/127.24.186.253@KRBTEST.COM (short username oryx)
I20260928 20:24:53.005300 31911 webserver.cc:460] Webserver started at http://127.24.186.253:40023/ using document root <none> and password file <none>
I20260928 20:24:53.005406 31911 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.005432 31911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.006057 31911 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:53.006584 32028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.006699 31911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:53.006762 31911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "45035f963ba871d333ae4795e284755f"
server_key_iv: "be6a80a4b0ff32d0081861614b0f13b9"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.006808 31911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.024071 31911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.025740 31911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.028915 31911 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.253:38315
I20260928 20:24:53.029145 32080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.253:38315 every 8 connection(s)
I20260928 20:24:53.029350 31911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
I20260928 20:24:53.030078 32081 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:24:53.030642 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Bootstrap starting.
I20260928 20:24:53.032990 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:24:53.033237 32081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: Bootstrap complete.
I20260928 20:24:53.033447 32081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } }
I20260928 20:24:53.033516 32081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f87c6b879e7c4a8ba3e71527cd60dbce, State: Initialized, Role: LEARNER
I20260928 20:24:53.033543 32081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } }
I20260928 20:24:53.033725 32081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:53.033720 32084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } }
I20260928 20:24:53.033825 32084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: LEARNER
I20260928 20:24:53.035208 32095 catalog_manager.cc:1259] Loaded cluster ID: 5280696460894460b2ce82cc9a75e73e
I20260928 20:24:53.035269 32095 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: loading cluster ID for follower catalog manager: success
I20260928 20:24:53.036262 32095 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: acquiring CA information for follower catalog manager: success
I20260928 20:24:53.036535 32095 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:53.047353 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.037852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.24.186.253:38315)
0928 20:24:53.039922 (+ 2070us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.039924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.039936 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:53.039965 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.039966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.039979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.040085 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.040693 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.041720 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.042479 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.042785 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.042869 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.044341 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.044343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.044343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.044350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.045989 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.046405 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.046406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.046407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.046476 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.046670 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.046671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.046671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.046916 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.047030 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.047139 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.047229 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2045,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.24.186.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.252.keytab.
Entry for principal oryx/127.24.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.252.keytab.
I20260928 20:24:53.074549 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:37251
--webserver_interface=127.24.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.252.keytab
--principal=oryx/127.24.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:53.157716 32099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:53.157953 32099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:53.157990 32099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:53.159225 32099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:53.159286 32099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:53.159312 32099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:53.159354 32099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:53.159382 32099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:53.160924 32099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.252.keytab
--principal=oryx/127.24.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:37251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:53.161199 32099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:53.161427 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.166393 32104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.166417 32107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.166636 32099 server_base.cc:1034] running on GCE node
W20260928 20:24:53.166393 32105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.166957 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.167212 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.168346 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790627093168335 us; error 28 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.170639 32099 init.cc:377] Logged in from keytab as oryx/127.24.186.252@KRBTEST.COM (short username oryx)
I20260928 20:24:53.171103 32099 webserver.cc:460] Webserver started at http://127.24.186.252:36239/ using document root <none> and password file <none>
I20260928 20:24:53.171340 32099 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.171414 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.171562 32099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:53.172708 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/instance:
uuid: "ca4c087324bc44eea2de6ae1e956029e"
format_stamp: "Formatted at 2026-09-28 20:24:53 on dist-test-slave-k477"
server_key: "b04f0a423ca249a3d18f1c2f6c3c323d"
server_key_iv: "70eab135b84e6778fc3939dbeb44a618"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.173076 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal/instance:
uuid: "ca4c087324bc44eea2de6ae1e956029e"
format_stamp: "Formatted at 2026-09-28 20:24:53 on dist-test-slave-k477"
server_key: "b04f0a423ca249a3d18f1c2f6c3c323d"
server_key_iv: "70eab135b84e6778fc3939dbeb44a618"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.176357 32099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 20:24:53.177217 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.177412 32099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:53.177522 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "ca4c087324bc44eea2de6ae1e956029e"
format_stamp: "Formatted at 2026-09-28 20:24:53 on dist-test-slave-k477"
server_key: "b04f0a423ca249a3d18f1c2f6c3c323d"
server_key_iv: "70eab135b84e6778fc3939dbeb44a618"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.177611 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.196382 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.198645 32099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:53.198813 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.203383 32099 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.252:37251
I20260928 20:24:53.203416 32166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.252:37251 every 8 connection(s)
I20260928 20:24:53.203785 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
I20260928 20:24:53.204867 32167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:53.206375 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:53.210302 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 32099
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.252@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:53.217473 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.207183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.24.186.254:33525)
0928 20:24:53.207283 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.207285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.207411 (+ 126us) server_negotiation.cc:407] Connection header received
0928 20:24:53.207923 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.207925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.207937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.208007 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.208643 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.209234 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.210272 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.210535 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.210626 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.214241 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.214243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.214243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.214251 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.215927 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.216459 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.216460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.216460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.216510 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.216792 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.216792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.216792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.216894 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.216944 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.217281 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.217369 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 20:24:53.217566 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.207067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38404)
0928 20:24:53.207252 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.207312 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.207785 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.208079 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.208083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.208338 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.208528 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.208532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.209346 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.209347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.210147 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.210149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.210660 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.210662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.210834 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.212150 (+ 1316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.212167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.214110 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.216007 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.216013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.216016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.216380 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.216574 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.216575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.216576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.216727 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.216961 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.216967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.217132 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.217229 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.217355 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
I20260928 20:24:53.218138 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:24:53.221175 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.211074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.24.186.252:37251)
0928 20:24:53.211253 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.211259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.211277 (+ 18us) server_negotiation.cc:407] Connection header received
0928 20:24:53.211767 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.211780 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.211925 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.212201 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.212844 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.213762 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.214644 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.215165 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.215226 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.218117 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.218120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.218120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.218125 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.219953 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.220360 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.220364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.220364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.220423 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.220632 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.220633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.220633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.220778 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.220946 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.220994 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.221074 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.252@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
I20260928 20:24:53.225633 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.218483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.24.186.253:38315)
0928 20:24:53.218545 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.218547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.218585 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:24:53.218678 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.218679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.218692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.218748 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.219089 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.219929 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.220914 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.221259 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.221360 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.223148 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.223149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.223149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.223157 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.224482 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.224835 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.224836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.224836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.224894 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.225100 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.225101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.225102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.225221 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.225295 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.225464 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.225544 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:24:53.225641 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.218425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:51010)
0928 20:24:53.218533 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.218545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.218611 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.218752 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.218753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.218952 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.219020 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.219022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.220032 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.220033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.220802 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.220805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.221359 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.221360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.221439 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.221795 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.221799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.223020 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.224569 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.224570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.224571 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.224747 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.224960 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.224961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.224961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.225017 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.225288 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.225288 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.225331 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.225434 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.225535 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 20:24:53.226173 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.252@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
I20260928 20:24:53.233434 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.226429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53720)
0928 20:24:53.226480 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.226493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.226534 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.226672 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.226673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.226810 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.226838 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.226839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.227496 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.227497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.228023 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.228025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.228403 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.228404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.228451 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.228859 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.228864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.230171 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.232289 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.232290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.232291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.232461 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.232641 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.232641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.232642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.232693 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.233099 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.233100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.233146 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.233214 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.233304 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:24:53.233443 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.226472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.24.186.252:37251)
0928 20:24:53.226525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.226527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.226535 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:53.226590 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.226591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.226602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.226671 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.226922 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.227407 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.228128 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.228321 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.228403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.230399 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.230400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.230400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.230406 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.232194 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.232531 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.232532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.232532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.232582 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.232882 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.232883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.232883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.233001 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.233063 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.233245 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.233350 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":37}
I20260928 20:24:53.235109 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Bootstrap starting.
I20260928 20:24:53.236073 32167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:53.236361 32167 log.cc:824] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:53.237217 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: No bootstrap required, opened a new log
I20260928 20:24:53.238898 32167 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } }
I20260928 20:24:53.239027 32167 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:53.239053 32167 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4c087324bc44eea2de6ae1e956029e, State: Initialized, Role: FOLLOWER
I20260928 20:24:53.239192 32167 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } }
I20260928 20:24:53.239523 32172 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } } }
I20260928 20:24:53.239527 32167 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:53.239665 32172 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:24:53.241477 32183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:53.241559 32183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:53.243832 32099 master_runner.cc:428] Detected that this master ca4c087324bc44eea2de6ae1e956029e is joining an existing cluster
I20260928 20:24:53.243884 32099 master_runner.cc:432] Initiating AddMaster RPC to add 127.24.186.252:37251
I20260928 20:24:53.251163 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.245264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.24.186.254:33525)
0928 20:24:53.245311 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.245313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.245324 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:53.245391 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.245391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.245402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.245451 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.245826 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.246542 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.247299 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.247515 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.247600 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.248640 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.248642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.248642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.248649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.250011 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.250357 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.250359 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.250359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.250415 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.250630 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.250631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.250632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.250746 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.250821 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.250985 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.251056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:53.251421 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.244846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38408)
0928 20:24:53.245281 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.245290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.245336 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.245465 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.245465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.245675 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.245748 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.245750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.246606 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.246607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.247213 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.247215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.247618 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.247619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.247678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.248193 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.248198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.248530 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.250117 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.250118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.250119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.250288 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.250488 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.250489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.250489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.250547 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.250812 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.250813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.250855 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.250946 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.251051 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":36,"spinlock_wait_cycles":113920,"thread_start_us":155,"threads_started":1}
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627092, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.254@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
I20260928 20:24:53.268810 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.253411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53732)
0928 20:24:53.253458 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.253468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.253501 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.253765 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.253766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.253933 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.253977 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.253979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.254710 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.254711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.255520 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.255522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.255969 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.255970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.256042 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.256397 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.256403 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.257802 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.266760 (+ 8958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.266763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.266764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.266999 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268052 (+ 1053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.268053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.268122 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268420 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.268421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.268471 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.268528 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.268661 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:24:53.268994 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.244850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.24.186.253:38315)
0928 20:24:53.244896 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.244898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.245120 (+ 222us) server_negotiation.cc:407] Connection header received
0928 20:24:53.245212 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.245214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.245225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.245257 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.245699 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.246481 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.247647 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.248037 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.248083 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.266189 (+ 18106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.266191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.266191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.266200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.267666 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268181 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.268182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.268183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.268244 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268469 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.268470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.268470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.268603 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.268644 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.268832 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.268911 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:24:53.268994 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.245188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.24.186.252:37251)
0928 20:24:53.245232 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.245235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.251549 (+ 6314us) server_negotiation.cc:407] Connection header received
0928 20:24:53.251646 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.251647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.251656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.251681 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.252032 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.252756 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.253968 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.254233 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.254299 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.265253 (+ 10954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.265256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.265256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.265264 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.267749 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268197 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.268198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.268198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.268276 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268466 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.268466 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.268467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.268580 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.268637 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.268820 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.268900 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:53.269325 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.245122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53730)
0928 20:24:53.251491 (+ 6369us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.251502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.251574 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.251726 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.251726 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.251899 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.251952 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.251954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.252979 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.252980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.253895 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.253897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.254311 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.254312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.254381 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.264523 (+ 10142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.264530 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.265073 (+ 543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.267867 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.267868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.267869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.268125 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268339 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.268340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.268405 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268666 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.268667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.268718 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.269087 (+ 369us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.269199 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6357,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260928 20:24:53.269399 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.244835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:51024)
0928 20:24:53.245036 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.245047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.245134 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.245329 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.245329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.245556 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.245622 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.245625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.246569 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.246570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.247534 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.247536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.248161 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.248162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.248225 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.264372 (+ 16147us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.264379 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.265353 (+ 974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.267775 (+ 2422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.267776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.267777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.268084 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268322 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.268323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.268393 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.268692 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.268693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.268721 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.269116 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.269250 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":46,"thread_start_us":150,"threads_started":1}
I20260928 20:24:53.270192 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.253480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.24.186.252:37251)
0928 20:24:53.253622 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.253624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.253631 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:53.253666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.253667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.253676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.253732 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.254045 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.254636 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.255615 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.255889 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.255967 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.257926 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.257927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.257928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.257934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.266644 (+ 8710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.267906 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.267907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.267908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.267969 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.268195 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.268195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.268196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.268346 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.269816 (+ 1470us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.269854 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.270084 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260928 20:24:53.270666 31447 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.24.186.252:37251: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e"
member_type: NON_VOTER
last_known_addr {
host: "127.24.186.252"
port: 37251
}
attrs {
promote: true
}
}
dest_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4"
cas_config_opid_index: 5
I20260928 20:24:53.270926 31447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } }
I20260928 20:24:53.271306 32007 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca4c087324bc44eea2de6ae1e956029e (127.24.186.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.272111 32141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:53.272472 32007 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.24.186.252:37251
I20260928 20:24:53.272550 32007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.272615 32007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.272694 32099 master.cc:537] Master@127.24.186.252:37251 shutting down...
I20260928 20:24:53.273105 32141 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 20:24:53.273619 32007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 20:24:53.274391 31431 consensus_peers.cc:487] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 -> Peer ca4c087324bc44eea2de6ae1e956029e (127.24.186.252:37251): Couldn't send request to peer ca4c087324bc44eea2de6ae1e956029e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 20:24:53.275166 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.271418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:51036)
0928 20:24:53.271451 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.271462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.271504 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.271672 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.271673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.271675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.271718 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.271719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.272568 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.272569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.273770 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.273773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.274884 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.274885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.274947 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.274967 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.275023 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.275083 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 20:24:53.275430 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.271506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.24.186.253:38315)
0928 20:24:53.271540 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.271542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.271549 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:53.271569 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.271569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.271590 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.271651 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.271776 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.272423 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.274185 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.274822 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.274888 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.274980 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.275332 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.275374 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:53.275687 32055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 LEARNER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 20:24:53.276034 32193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 20:24:53.276703 32099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:53.276795 32099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:53.276846 32099 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: stopping tablet replica
I20260928 20:24:53.277040 32055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca4c087324bc44eea2de6ae1e956029e (127.24.186.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.278097 32084 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cf5c1738184d959f42e8f4d3ac8ec4. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.278174 32084 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: LEARNER
I20260928 20:24:53.278126 32195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: NON_VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: true } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.278236 32195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: LEARNER
I20260928 20:24:53.279379 32193 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: attempting to promote NON_VOTER f87c6b879e7c4a8ba3e71527cd60dbce to VOTER
I20260928 20:24:53.279616 32193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } }
I20260928 20:24:53.280129 32055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 LEARNER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260928 20:24:53.280164 31400 connection.cc:535] client connection to 127.24.186.252:37251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:24:53.280354 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.280103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53744)
0928 20:24:53.280285 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.280326 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 20:24:53.280373 32193 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 20:24:53.280773 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.280688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53750)
0928 20:24:53.280726 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.280748 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:37251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 20:24:53.280891 31431 consensus_peers.cc:487] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 -> Peer ca4c087324bc44eea2de6ae1e956029e (127.24.186.252:37251): Couldn't send request to peer ca4c087324bc44eea2de6ae1e956029e. Status: Network error: Client connection negotiation failed: client connection to 127.24.186.252:37251: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 20:24:53.281087 32006 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.281929 32006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.282011 32006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.281965 32193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.282090 32193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.282176 32055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.282908 32195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.283032 32195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:53.292636 32099 master.cc:552] Master@127.24.186.252:37251 shutdown complete.
I20260928 20:24:53.292713 32099 master_runner.cc:305] Clearing existing system tablet
I20260928 20:24:53.292912 32099 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.292984 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.293907 32099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:24:53.294843 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.295111 32099 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.295188 32099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:53.295217 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "ca4c087324bc44eea2de6ae1e956029e"
format_stamp: "Formatted at 2026-09-28 20:24:53 on dist-test-slave-k477"
server_key: "b04f0a423ca249a3d18f1c2f6c3c323d"
server_key_iv: "70eab135b84e6778fc3939dbeb44a618"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.296233 32099 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:53.297883 32099 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 20:24:53.297960 32099 log.cc:1196] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 20:24:53.298168 32099 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Deleting consensus metadata
I20260928 20:24:53.298296 32099 master_runner.cc:315] Copying system tablet from 127.24.186.254:33525
I20260928 20:24:53.298941 32099 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: tablet copy: Beginning tablet copy session from remote peer at address 127.24.186.254:33525
I20260928 20:24:53.306061 32203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.299402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38420)
0928 20:24:53.299553 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.299564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.299644 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.299837 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.299837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.300049 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.300141 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.300144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.300986 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.300987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.301828 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.301830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.302220 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.302222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.302292 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.302745 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.302750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.303160 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.304893 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.304894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.304895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.305117 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.305316 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.305317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.305317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.305383 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.305708 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.305709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.305756 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.305837 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.305945 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260928 20:24:53.306113 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.299508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.24.186.254:33525)
0928 20:24:53.299563 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.299565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.299627 (+ 62us) server_negotiation.cc:407] Connection header received
0928 20:24:53.299715 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.299716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.299725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.299818 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.300243 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.300899 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.301908 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.302111 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.302220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.303273 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.303275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.303276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.303282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.304815 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.305191 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.305192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.305192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.305252 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.305506 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.305507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.305508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.305633 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.305704 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.305895 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.305973 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:24:53.306526 31467 tablet_copy_service.cc:140] P 37cf5c1738184d959f42e8f4d3ac8ec4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ca4c087324bc44eea2de6ae1e956029e ({username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:38420)
I20260928 20:24:53.306592 31467 tablet_copy_service.cc:161] P 37cf5c1738184d959f42e8f4d3ac8ec4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ca4c087324bc44eea2de6ae1e956029e at {username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:38420: session id = ca4c087324bc44eea2de6ae1e956029e-00000000000000000000000000000000
I20260928 20:24:53.307092 31467 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 20:24:53.307678 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 20:24:53.309080 32099 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: tablet copy: Starting download of 0 data blocks...
I20260928 20:24:53.309252 32099 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: tablet copy: Starting download of 1 WAL segments...
I20260928 20:24:53.310276 32099 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 20:24:53.311563 31467 tablet_copy_service.cc:342] P 37cf5c1738184d959f42e8f4d3ac8ec4: Request end of tablet copy session ca4c087324bc44eea2de6ae1e956029e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:38420
I20260928 20:24:53.311656 31467 tablet_copy_service.cc:433] P 37cf5c1738184d959f42e8f4d3ac8ec4: ending tablet copy session ca4c087324bc44eea2de6ae1e956029e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ca4c087324bc44eea2de6ae1e956029e
I20260928 20:24:53.314538 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.316031 32204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.316068 32099 server_base.cc:1034] running on GCE node
W20260928 20:24:53.316076 32205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.316291 32207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.316452 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.316601 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.317747 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790627093317752 us; error 39 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.319099 32099 init.cc:377] Logged in from keytab as oryx/127.24.186.252@KRBTEST.COM (short username oryx)
I20260928 20:24:53.319406 32099 webserver.cc:460] Webserver started at http://127.24.186.252:44589/ using document root <none> and password file <none>
I20260928 20:24:53.319540 32099 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.319593 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.320313 32099 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.321005 32214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.321321 32099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:53.321388 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "ca4c087324bc44eea2de6ae1e956029e"
format_stamp: "Formatted at 2026-09-28 20:24:53 on dist-test-slave-k477"
server_key: "b04f0a423ca249a3d18f1c2f6c3c323d"
server_key_iv: "70eab135b84e6778fc3939dbeb44a618"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.321457 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.330101 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.331986 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.335644 32099 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.252:37251
I20260928 20:24:53.336079 32266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.252:37251 every 8 connection(s)
I20260928 20:24:53.336120 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
I20260928 20:24:53.336834 32267 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:24:53.337281 32267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Bootstrap starting.
I20260928 20:24:53.340075 32267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:24:53.340322 32267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Bootstrap complete.
I20260928 20:24:53.340535 32267 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } }
I20260928 20:24:53.340631 32267 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca4c087324bc44eea2de6ae1e956029e, State: Initialized, Role: LEARNER
I20260928 20:24:53.340682 32267 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } }
I20260928 20:24:53.340878 32267 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:53.342162 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.342288 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: LEARNER
I20260928 20:24:53.342461 32281 catalog_manager.cc:1259] Loaded cluster ID: 5280696460894460b2ce82cc9a75e73e
I20260928 20:24:53.342506 32281 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: loading cluster ID for follower catalog manager: success
I20260928 20:24:53.343462 32281 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: acquiring CA information for follower catalog manager: success
I20260928 20:24:53.343752 32281 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:24:53.388232 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.381285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.24.186.252:37251)
0928 20:24:53.381396 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.381398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.381411 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:24:53.381438 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.381439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.381450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.381532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.382117 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.382947 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.383771 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.384042 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.384131 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.385486 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.385488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.385488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.385497 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.386931 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.387316 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.387317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.387318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.387384 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.387614 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.387615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.387615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.387741 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.387880 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.387989 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.388088 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260928 20:24:53.388839 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31509
I20260928 20:24:53.395488 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:38787
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:33525
--webserver_port=39291
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:53.483680 32283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:53.483934 32283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:53.483963 32283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:53.485738 32283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:53.485816 32283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:53.485854 32283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:53.488111 32283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:38787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=39291
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:53.488394 32283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:53.488654 32283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.494108 32289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.494110 32291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.494339 32288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.494603 32283 server_base.cc:1034] running on GCE node
I20260928 20:24:53.495018 32283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.495463 32283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.496668 32283 hybrid_clock.cc:648] HybridClock initialized: now 1790627093496612 us; error 119 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.499358 32283 init.cc:377] Logged in from keytab as oryx/127.24.186.193@KRBTEST.COM (short username oryx)
I20260928 20:24:53.500007 32283 webserver.cc:460] Webserver started at http://127.24.186.193:39291/ using document root <none> and password file <none>
I20260928 20:24:53.500217 32283 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.500308 32283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.502068 32283 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:53.502930 32298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.503136 32283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:53.503261 32283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "c456a93ad7bf4c76bdc5c6045231d875"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "7c87200daf594fd911009695565940f0"
server_key_iv: "bdd9ed21b5c2fe32b097dcff85180a08"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.503620 32283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.519325 32283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.521685 32283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:53.521860 32283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.522123 32283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:53.522511 32283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:53.522576 32283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.522619 32283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:53.522650 32283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.528885 32283 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:38787
I20260928 20:24:53.528934 32411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:38787 every 8 connection(s)
I20260928 20:24:53.529311 32283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:24:53.530969 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 32283
I20260928 20:24:53.531153 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31643
I20260928 20:24:53.537333 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:42867
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:33525
--webserver_port=45109
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
I20260928 20:24:53.545079 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.530390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47765 (local address 127.24.186.252:37251)
0928 20:24:53.530431 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.530433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.532654 (+ 2221us) server_negotiation.cc:407] Connection header received
0928 20:24:53.532749 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.532750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.532765 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.532795 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.533529 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.534383 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.535545 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.536014 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.536092 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.541569 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.541571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.541572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.541580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.543300 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.543874 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.543876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.543876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.543938 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544254 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.544255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.544256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.544408 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.544461 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.544849 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.544954 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:24:53.545629 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.529984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:34541 (local address 127.24.186.253:38315)
0928 20:24:53.530040 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.530043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.535938 (+ 5895us) server_negotiation.cc:407] Connection header received
0928 20:24:53.535967 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.535968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.535981 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.536031 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.536473 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.537043 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.539981 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.540317 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.540398 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.542777 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.542779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.542780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.542789 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.544411 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544835 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.544836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.544836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.544894 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.545088 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.545089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.545089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.545233 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.545279 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.545475 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.545550 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:24:53.545686 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.530302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.24.186.193:47765)
0928 20:24:53.532604 (+ 2302us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.532612 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.532678 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.532854 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.532855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.533082 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.533405 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.533411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.534475 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.534475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.535455 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.535458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.536153 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.536154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.536270 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.536804 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.536814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.541453 (+ 4639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.543396 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.543406 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.543408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.543752 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.544017 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.544018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.544157 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.544493 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.544500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.544677 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.545234 (+ 557us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.545436 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2277,"mutex_wait_us":41,"thread_start_us":55,"threads_started":1}
I20260928 20:24:53.545935 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.530015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.24.186.193:34541)
0928 20:24:53.532460 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.532471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.532520 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.536113 (+ 3593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.536114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.536312 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.536378 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.536380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.537144 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.537145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.539827 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.539834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.540416 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.540417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.540495 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.540999 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.541005 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.542653 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.544492 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.544493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.544751 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.544968 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.544969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.545023 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.545326 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.545327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.545364 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.545729 (+ 365us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.545839 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2417,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260928 20:24:53.546319 32414 heartbeater.cc:347] Connected to a master server at 127.24.186.253:38315
I20260928 20:24:53.546404 32414 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.546571 32414 heartbeater.cc:511] Master 127.24.186.253:38315 requested a full tablet report, sending...
I20260928 20:24:53.546278 32412 heartbeater.cc:347] Connected to a master server at 127.24.186.252:37251
I20260928 20:24:53.546752 32412 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.546813 32412 heartbeater.cc:511] Master 127.24.186.252:37251 requested a full tablet report, sending...
I20260928 20:24:53.546875 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.530414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47373 (local address 127.24.186.254:33525)
0928 20:24:53.530469 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.530471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.530481 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:53.532385 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.532387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.532401 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.532453 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.533519 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.534114 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.535577 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.535790 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.535903 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.541181 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.541184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.541184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.541194 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.544255 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544742 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.544744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.544744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.544810 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.545035 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.545036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.545036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.545201 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.545257 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.546645 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.546739 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:24:53.547147 32231 ts_manager.cc:194] Registered new tserver with Master: c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:53.547201 32045 ts_manager.cc:194] Registered new tserver with Master: c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:53.548072 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.530014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:47373)
0928 20:24:53.530207 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.530251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.532255 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.532602 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.532607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.533073 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.533405 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.533411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.534227 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.534228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.535455 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.535458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.535977 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.535978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.536145 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.536970 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.536975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.541046 (+ 4071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.544361 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.544363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.544643 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.544889 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.544890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.544891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.544960 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.545571 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.545572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.545624 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.547538 (+ 1914us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.547667 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":8,"thread_start_us":119,"threads_started":1}
I20260928 20:24:53.548463 32413 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:53.548539 32413 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.548611 32413 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:53.548861 31447 ts_manager.cc:194] Re-registered known tserver with Master: c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:53.549249 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:47373
W20260928 20:24:53.626355 32419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:53.626580 32419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:53.626606 32419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:53.628100 32419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:53.628165 32419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:53.628197 32419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:53.631454 32419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:42867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=45109
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:53.631786 32419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:53.632102 32419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.637449 32424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.637434 32425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.637492 32427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.637728 32419 server_base.cc:1034] running on GCE node
I20260928 20:24:53.637925 32419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.638191 32419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.639351 32419 hybrid_clock.cc:648] HybridClock initialized: now 1790627093639337 us; error 27 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.642099 32419 init.cc:377] Logged in from keytab as oryx/127.24.186.194@KRBTEST.COM (short username oryx)
I20260928 20:24:53.642563 32419 webserver.cc:460] Webserver started at http://127.24.186.194:45109/ using document root <none> and password file <none>
I20260928 20:24:53.642841 32419 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.642926 32419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.644397 32419 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:53.645076 32434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.645282 32419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:53.645382 32419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "7f43d62f60d6443bb7e4a47d028df299"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "6d1c3e8f8a9327e18fe8b6e100c9e3c1"
server_key_iv: "8b758f772dfb6503ce5ea05ac36ca7ac"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.645779 32419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.662422 32419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.665187 32419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:53.665352 32419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.665601 32419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:53.665933 32419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:53.665990 32419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.666034 32419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:53.666060 32419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.672331 32419 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:42867
I20260928 20:24:53.672418 32547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:42867 every 8 connection(s)
I20260928 20:24:53.672772 32419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:53.674285 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 32419
I20260928 20:24:53.674633 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31777
I20260928 20:24:53.679557 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:45335
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:33525
--webserver_port=38495
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:53.690910 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.673613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:42363 (local address 127.24.186.253:38315)
0928 20:24:53.673725 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.673727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.675419 (+ 1692us) server_negotiation.cc:407] Connection header received
0928 20:24:53.679963 (+ 4544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.679966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.679980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.680024 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.680984 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.681688 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683185 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683487 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683531 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.687034 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.687036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.687036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.687046 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.689281 (+ 2235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.689810 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.689813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.689813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.689875 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.690166 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.690166 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.690167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.690307 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.690347 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.690730 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.690814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 20:24:53.691421 32553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.673583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.24.186.194:42363)
0928 20:24:53.674318 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.674361 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.679747 (+ 5386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.680129 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.680133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.680815 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.680887 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.680889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.681773 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.681774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683073 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.683076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683626 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683769 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.684680 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.684684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.686893 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.689376 (+ 2483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.689385 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.689387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.689705 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.689945 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.689946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.689946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.690090 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.690390 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.690395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.690563 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.690998 (+ 435us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.691192 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":57,"thread_start_us":542,"threads_started":1}
I20260928 20:24:53.691645 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.673668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:56775 (local address 127.24.186.254:33525)
0928 20:24:53.674193 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.674197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.674601 (+ 404us) server_negotiation.cc:407] Connection header received
0928 20:24:53.681068 (+ 6467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.681070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.681086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.681120 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.681562 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.682349 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683168 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683483 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683551 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.687727 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.687729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.687730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.687740 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.690257 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.690694 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.690696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.690696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.690760 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.690998 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.690999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.690999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.691165 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.691215 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.691448 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.691536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260928 20:24:53.692086 32554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.673583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:56775)
0928 20:24:53.674534 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.674543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.680975 (+ 6432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.681192 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.681193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.681396 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.681456 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.681458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.682454 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.682454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683069 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.683071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683619 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683769 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.684484 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.684495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.687601 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.690361 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.690362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.690363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.690595 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.690855 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.690856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.690857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.690918 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.691269 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.691270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.691317 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.691768 (+ 451us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.691922 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":76,"threads_started":1}
I20260928 20:24:53.692484 32549 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:53.692745 32549 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.693580 32549 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:53.694043 31447 ts_manager.cc:194] Re-registered known tserver with Master: 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
I20260928 20:24:53.694594 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:56775
I20260928 20:24:53.698601 32550 heartbeater.cc:347] Connected to a master server at 127.24.186.253:38315
I20260928 20:24:53.698674 32550 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.698770 32550 heartbeater.cc:511] Master 127.24.186.253:38315 requested a full tablet report, sending...
I20260928 20:24:53.699440 32045 ts_manager.cc:194] Registered new tserver with Master: 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
I20260928 20:24:53.700558 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.674738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40215 (local address 127.24.186.252:37251)
0928 20:24:53.675290 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.675293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.679490 (+ 4197us) server_negotiation.cc:407] Connection header received
0928 20:24:53.680060 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.680061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.680073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.680102 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.680775 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.681538 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683193 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683492 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683531 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.696219 (+ 12688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.696221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.696221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.696230 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.699214 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.699708 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.699709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.699710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.699774 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.700000 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.700001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.700002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.700135 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.700176 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.700375 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.700461 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527}
I20260928 20:24:53.700886 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.674411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.24.186.194:40215)
0928 20:24:53.679400 (+ 4989us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.679413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.679786 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.680171 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.680171 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.680456 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.680678 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.680682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.681657 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.681658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683074 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.683076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.683624 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.683625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.683769 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.684611 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.684619 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.687242 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.699317 (+ 12075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.699318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.699319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.699620 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.699864 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.699865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.699865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.699931 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.700220 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.700221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.700264 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.700644 (+ 380us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.700783 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4967,"mutex_wait_us":62,"thread_start_us":61,"threads_started":1}
I20260928 20:24:53.701411 32548 heartbeater.cc:347] Connected to a master server at 127.24.186.252:37251
I20260928 20:24:53.701479 32548 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.701529 32548 heartbeater.cc:511] Master 127.24.186.252:37251 requested a full tablet report, sending...
I20260928 20:24:53.701861 32231 ts_manager.cc:194] Registered new tserver with Master: 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
W20260928 20:24:53.774520 32555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:53.774791 32555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:53.774830 32555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:53.776046 32555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:53.776125 32555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:53.776177 32555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:53.777773 32555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:45335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=38495
--tserver_master_addrs=127.24.186.254:33525,127.24.186.253:38315,127.24.186.252:37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:53.778069 32555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:53.778323 32555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:53.783507 32561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.783545 32560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:53.783560 32563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:53.783586 32555 server_base.cc:1034] running on GCE node
I20260928 20:24:53.784067 32555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:53.784317 32555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:53.785521 32555 hybrid_clock.cc:648] HybridClock initialized: now 1790627093785499 us; error 44 us; skew 500 ppm
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:53.789474 32555 init.cc:377] Logged in from keytab as oryx/127.24.186.195@KRBTEST.COM (short username oryx)
I20260928 20:24:53.790347 32555 webserver.cc:460] Webserver started at http://127.24.186.195:38495/ using document root <none> and password file <none>
I20260928 20:24:53.790649 32555 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:53.790730 32555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:53.792270 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.788426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53780)
0928 20:24:53.788534 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.788546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.788617 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.788931 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.788933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.788938 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.788996 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.788998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.789906 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.789908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.791291 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.791299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.791986 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.791987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.792063 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.792069 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.792138 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.792198 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":46,"threads_started":1}
I20260928 20:24:53.792289 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.788517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.24.186.252:37251)
0928 20:24:53.788560 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.788562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.788612 (+ 50us) server_negotiation.cc:407] Connection header received
0928 20:24:53.788688 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.788689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.788714 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.788806 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.789134 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.789742 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.791402 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.791896 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.791965 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.792011 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.792151 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.792216 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:24:53.792603 32555 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:24:53.792599 32241 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 20:24:53.792759 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37cf5c1738184d959f42e8f4d3ac8ec4. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.793063 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: LEARNER
I20260928 20:24:53.793437 32572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.793622 32555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:53.793728 32555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "d76b21af060c44dc82dd9ce32f61981e"
format_stamp: "Formatted at 2026-09-28 20:24:52 on dist-test-slave-k477"
server_key: "c90997576e0fb35249282f891037c799"
server_key_iv: "af110a0433147f1ce07538c87408ffbb"
server_key_version: "encryptionkey@0"
I20260928 20:24:53.794384 32555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:53.812541 32555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:53.816056 32555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:53.816250 32555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:53.816510 32555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:24:53.816855 32555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:53.816931 32555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.816975 32555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:53.816989 32555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:53.823163 32555 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45335
I20260928 20:24:53.823328 32685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45335 every 8 connection(s)
I20260928 20:24:53.823623 32555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:53.828094 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 32555
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.253@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.252@KRBTEST.COM
Sep 28 20:24:53 dist-test-slave-k477 krb5kdc[31402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627093, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:53.837931 32690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.24.186.195:60111)
0928 20:24:53.824631 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.824682 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.825940 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.826420 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.826421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.827038 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.827159 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.827162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.828083 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.828084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.829461 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.829464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.829981 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.830128 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.830853 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.830858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.833502 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.836110 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.836121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.836458 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.836718 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.836720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.836888 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.837184 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.837192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.837373 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.837490 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.837682 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":155,"thread_start_us":110,"threads_started":1}
I20260928 20:24:53.838404 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:60111 (local address 127.24.186.252:37251)
0928 20:24:53.824487 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.824489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.824776 (+ 287us) server_negotiation.cc:407] Connection header received
0928 20:24:53.826144 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.826147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.826168 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.826238 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.827271 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.827999 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829581 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.829839 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829909 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.833611 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.833613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.833613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.833622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.835642 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836535 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.836536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.836537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.836604 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836963 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.836964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.836964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.837112 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.837182 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.838226 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.838312 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:24:53.838513 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:34495 (local address 127.24.186.253:38315)
0928 20:24:53.824537 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.824539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.824776 (+ 237us) server_negotiation.cc:407] Connection header received
0928 20:24:53.826090 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.826092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.826107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.826170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.827035 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.827790 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.830030 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.830331 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.830396 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.834069 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.834070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.834071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.834078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.836387 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836728 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.836730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.836730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.836784 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.837030 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.837031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.837031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.837175 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.837235 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.838335 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.838411 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:24:53.838518 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:59837)
0928 20:24:53.824930 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.824939 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.826066 (+ 1127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.826420 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.826421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.827038 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.827159 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.827162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.827943 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.827944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.829463 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.829466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.829979 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.830129 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.830803 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.830815 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.834138 (+ 3323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.836836 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.836837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.837079 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.837312 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.837313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.837314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.837370 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.838110 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.838111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.838175 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.838250 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.838373 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":44,"thread_start_us":63,"threads_started":1}
I20260928 20:24:53.838975 32686 heartbeater.cc:347] Connected to a master server at 127.24.186.252:37251
I20260928 20:24:53.838999 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:59837 (local address 127.24.186.254:33525)
0928 20:24:53.824860 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.824862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.824988 (+ 126us) server_negotiation.cc:407] Connection header received
0928 20:24:53.826240 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.826242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.826256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.826325 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.827274 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.827763 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829581 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.829890 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.829971 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.834228 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.834231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.834231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.834239 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.836757 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.837192 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.837193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.837193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.837248 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.837441 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.837443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.837443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.837993 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.838077 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.838806 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.838893 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:24:53.839071 32686 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.839395 32686 heartbeater.cc:511] Master 127.24.186.252:37251 requested a full tablet report, sending...
I20260928 20:24:53.839478 32687 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33525
I20260928 20:24:53.839566 32687 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.839636 32687 heartbeater.cc:511] Master 127.24.186.254:33525 requested a full tablet report, sending...
I20260928 20:24:53.839947 32231 ts_manager.cc:194] Registered new tserver with Master: d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335)
I20260928 20:24:53.839947 31447 ts_manager.cc:194] Re-registered known tserver with Master: d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335)
I20260928 20:24:53.840416 31447 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:59837
I20260928 20:24:53.840909 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.824446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.24.186.195:34495)
0928 20:24:53.824631 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.824683 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.825940 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.826362 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.826367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.826702 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:53.826915 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.826920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.827940 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.827941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.829461 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.829464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.830449 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.830450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.830529 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.831034 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:53.831038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:53.833975 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:53.836470 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.836471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.836671 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.836872 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:53.836873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:53.836873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:53.836959 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:53.837259 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:53.837260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:53.837373 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.840724 (+ 3351us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.840832 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":127,"thread_start_us":99,"threads_started":1}
I20260928 20:24:53.841499 32688 heartbeater.cc:347] Connected to a master server at 127.24.186.253:38315
I20260928 20:24:53.841553 32688 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:53.841624 32688 heartbeater.cc:511] Master 127.24.186.253:38315 requested a full tablet report, sending...
I20260928 20:24:53.841910 32045 ts_manager.cc:194] Registered new tserver with Master: d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335)
I20260928 20:24:53.842862 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.829815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.24.186.254:33525)
0928 20:24:53.830414 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.830416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.830424 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:53.830446 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.830447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.830460 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.830528 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.831600 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.832234 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.833507 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.833813 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.839213 (+ 5400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.839280 (+ 67us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.839281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.839281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.839286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.841162 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.841625 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.841626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.841627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.841684 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.842158 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.842159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.842159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.842317 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.842417 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.842546 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.842742 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":65,"threads_started":1}
I20260928 20:24:53.844959 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.831066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.24.186.252:37251)
0928 20:24:53.831691 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.831693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.832392 (+ 699us) server_negotiation.cc:407] Connection header received
0928 20:24:53.832490 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.832491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.832504 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.832549 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.833274 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.834019 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.835361 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.835720 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.835764 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.841468 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.841469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.841470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.841478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.843565 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.844108 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.844110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.844110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.844197 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.844403 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.844404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.844404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.844549 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.844657 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.844766 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.844843 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":70,"threads_started":1}
I20260928 20:24:53.846690 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.830937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.24.186.253:38315)
0928 20:24:53.831640 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.831642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.832400 (+ 758us) server_negotiation.cc:407] Connection header received
0928 20:24:53.832626 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.832627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.832639 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.832686 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:53.835249 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.836028 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.841807 (+ 5779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.842181 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.842249 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.844051 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:53.844052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:53.844053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:53.844061 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:53.845642 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.846014 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.846014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.846014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.846062 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:53.846260 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:53.846260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:53.846261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:53.846361 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:53.846429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.846562 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.846624 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":56,"threads_started":1}
I20260928 20:24:53.844704 31447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:53.847162 31447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:53.847981 32566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: NON_VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: true } } }
I20260928 20:24:53.848114 32566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.849330 32704 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: attempting to promote NON_VOTER ca4c087324bc44eea2de6ae1e956029e to VOTER
I20260928 20:24:53.849458 32704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } }
I20260928 20:24:53.849828 32241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 LEARNER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 20:24:53.850018 32566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 20:24:53.850245 32055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Refusing update from remote peer 37cf5c1738184d959f42e8f4d3ac8ec4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 20:24:53.850433 32566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 20:24:53.850689 32704 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ca4c087324bc44eea2de6ae1e956029e (127.24.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.850914 32241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ca4c087324bc44eea2de6ae1e956029e (127.24.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.851173 32055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ca4c087324bc44eea2de6ae1e956029e (127.24.186.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.851722 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.851788 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.851835 32270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.851871 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.851907 32270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:53.851928 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.852010 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.852063 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:53.853502 32717 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:53.853619 32717 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:53.857908 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.853438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.24.186.195:45335)
0928 20:24:53.854015 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.854020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.854032 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:53.854110 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.854112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.854131 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.854218 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.854412 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855447 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.856868 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857538 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857628 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.857675 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.857777 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.857824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":95,"threads_started":1}
I20260928 20:24:53.857915 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.852886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45335 (local address 127.0.0.1:38634)
0928 20:24:53.853096 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.853113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.853186 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.854293 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.854294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.854296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.854352 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.854355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855543 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.855544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.856783 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.856786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857622 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.857687 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.857692 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.857760 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.857811 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260928 20:24:53.858083 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.852861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38787 (local address 127.0.0.1:35214)
0928 20:24:53.853001 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.853012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.853055 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.854871 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.854872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.854874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.854908 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.854909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855902 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.855903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.856972 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.856974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857872 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.857933 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.857936 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.857967 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.858018 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 20:24:53.857939 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.853403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.24.186.194:42867)
0928 20:24:53.854196 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.854201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.854324 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:24:53.854463 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.854466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.854488 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.854577 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.854722 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855700 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.856693 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857430 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.857592 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.857824 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.857865 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":86,"threads_started":1}
I20260928 20:24:53.858510 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.853083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:42867 (local address 127.0.0.1:57256)
0928 20:24:53.854262 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.854271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.854337 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.854578 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.854579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.854581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.854630 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.854632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855782 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.855783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.856579 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.856581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857649 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.857725 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.857730 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.858379 (+ 649us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.858431 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"mutex_wait_us":37,"thread_start_us":63,"threads_started":1}
I20260928 20:24:53.858536 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.853147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.24.186.193:38787)
0928 20:24:53.854701 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.854706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.854715 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:53.854754 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.854756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.854770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.854862 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.854961 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.855758 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.857127 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.857788 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.858295 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.858382 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.858411 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.858452 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1514,"thread_start_us":70,"threads_started":1}
I20260928 20:24:53.858863 32620 tablet_service.cc:1467] Processing CreateTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb (DEFAULT_TABLE table=test-table [id=62adae8b44dc4a06b113a6c050706995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:53.859109 32346 tablet_service.cc:1467] Processing CreateTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb (DEFAULT_TABLE table=test-table [id=62adae8b44dc4a06b113a6c050706995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:53.859115 32482 tablet_service.cc:1467] Processing CreateTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb (DEFAULT_TABLE table=test-table [id=62adae8b44dc4a06b113a6c050706995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:53.859258 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ff12535c9b4129bae439fbf8e9d7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:53.859409 32482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ff12535c9b4129bae439fbf8e9d7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:53.859483 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ff12535c9b4129bae439fbf8e9d7fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:53.862284 32720 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Bootstrap starting.
I20260928 20:24:53.862756 32722 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Bootstrap starting.
I20260928 20:24:53.863081 32720 tablet_bootstrap.cc:654] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:53.863461 32722 tablet_bootstrap.cc:654] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:53.863514 32720 log.cc:824] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:53.863788 32722 log.cc:824] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:53.863987 32721 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Bootstrap starting.
I20260928 20:24:53.864455 32720 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: No bootstrap required, opened a new log
I20260928 20:24:53.864535 32720 ts_tablet_manager.cc:1397] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:53.864595 32722 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: No bootstrap required, opened a new log
I20260928 20:24:53.864601 32721 tablet_bootstrap.cc:654] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:53.864665 32722 ts_tablet_manager.cc:1397] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:24:53.864919 32721 log.cc:824] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:53.866159 32721 tablet_bootstrap.cc:492] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: No bootstrap required, opened a new log
I20260928 20:24:53.866287 32721 ts_tablet_manager.cc:1397] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:53.866433 32722 raft_consensus.cc:348] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.866595 32722 raft_consensus.cc:374] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:53.866638 32722 raft_consensus.cc:729] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f43d62f60d6443bb7e4a47d028df299, State: Initialized, Role: FOLLOWER
I20260928 20:24:53.866765 32722 consensus_queue.cc:260] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.867062 32549 heartbeater.cc:503] Master 127.24.186.254:33525 was elected leader, sending a full tablet report...
I20260928 20:24:53.868031 32722 ts_tablet_manager.cc:1428] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:24:53.869397 32720 raft_consensus.cc:348] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.869590 32720 raft_consensus.cc:374] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:53.869629 32720 raft_consensus.cc:729] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c456a93ad7bf4c76bdc5c6045231d875, State: Initialized, Role: FOLLOWER
I20260928 20:24:53.869659 32721 raft_consensus.cc:348] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.869755 32720 consensus_queue.cc:260] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.869921 32721 raft_consensus.cc:374] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:53.869966 32721 raft_consensus.cc:729] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76b21af060c44dc82dd9ce32f61981e, State: Initialized, Role: FOLLOWER
I20260928 20:24:53.870090 32721 consensus_queue.cc:260] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.870186 32720 ts_tablet_manager.cc:1428] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 20:24:53.870303 32413 heartbeater.cc:503] Master 127.24.186.254:33525 was elected leader, sending a full tablet report...
I20260928 20:24:53.870340 32721 ts_tablet_manager.cc:1428] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 20:24:53.870510 32687 heartbeater.cc:503] Master 127.24.186.254:33525 was elected leader, sending a full tablet report...
W20260928 20:24:53.929752 32551 tablet.cc:2369] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:53.993717 32727 raft_consensus.cc:482] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:53.993891 32727 raft_consensus.cc:504] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.994305 32727 leader_election.cc:288] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335), 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
I20260928 20:24:53.997442 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.994801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:44043 (local address 127.24.186.195:45335)
0928 20:24:53.994854 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.994856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.994868 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:24:53.994968 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.994969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.994982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.995044 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.995234 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.995730 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.996742 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.997089 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.997183 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.997193 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.997333 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.997369 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:24:53.997442 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.994683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45335 (local address 127.24.186.193:44043)
0928 20:24:53.994801 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.994815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.994880 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.995071 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.995072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.995074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.995155 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.995156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.995848 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.995849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.996648 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.996650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.997182 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.997182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.997259 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.997263 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.997328 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.997369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 20:24:53.997534 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.994942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48665 (local address 127.24.186.194:42867)
0928 20:24:53.994999 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:53.995002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:53.995012 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:53.995040 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:53.995042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:53.995051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:53.995140 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:53.995300 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.995769 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.996793 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.997158 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.997245 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.997253 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:53.997407 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 20:24:53.997447 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:24:53.997575 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:53.994866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:42867 (local address 127.24.186.193:48665)
0928 20:24:53.994922 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:53.994934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:53.994975 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:53.995144 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:53.995145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:53.995147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:53.995222 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.995224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.995907 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.995909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.996679 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:53.996682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:53.997244 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:53.997245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:53.997322 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:53.997325 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:53.997399 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:24:53.997444 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:24:53.997898 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ff12535c9b4129bae439fbf8e9d7fb" candidate_uuid: "c456a93ad7bf4c76bdc5c6045231d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f43d62f60d6443bb7e4a47d028df299" is_pre_election: true
I20260928 20:24:53.997957 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ff12535c9b4129bae439fbf8e9d7fb" candidate_uuid: "c456a93ad7bf4c76bdc5c6045231d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76b21af060c44dc82dd9ce32f61981e" is_pre_election: true
I20260928 20:24:53.998072 32640 raft_consensus.cc:2454] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c456a93ad7bf4c76bdc5c6045231d875 in term 0.
I20260928 20:24:53.998072 32502 raft_consensus.cc:2454] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c456a93ad7bf4c76bdc5c6045231d875 in term 0.
I20260928 20:24:53.998314 32299 leader_election.cc:302] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f43d62f60d6443bb7e4a47d028df299, c456a93ad7bf4c76bdc5c6045231d875; no voters:
I20260928 20:24:53.998536 32727 raft_consensus.cc:2781] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:53.998596 32727 raft_consensus.cc:482] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:53.998624 32727 raft_consensus.cc:3037] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:53.999485 32727 raft_consensus.cc:504] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:53.999655 32727 leader_election.cc:288] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [CANDIDATE]: Term 1 election: Requested vote from peers d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335), 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867)
I20260928 20:24:53.999879 32640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ff12535c9b4129bae439fbf8e9d7fb" candidate_uuid: "c456a93ad7bf4c76bdc5c6045231d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76b21af060c44dc82dd9ce32f61981e"
I20260928 20:24:53.999912 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ff12535c9b4129bae439fbf8e9d7fb" candidate_uuid: "c456a93ad7bf4c76bdc5c6045231d875" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f43d62f60d6443bb7e4a47d028df299"
I20260928 20:24:53.999986 32640 raft_consensus.cc:3037] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:54.000022 32502 raft_consensus.cc:3037] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:54.000980 32502 raft_consensus.cc:2454] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c456a93ad7bf4c76bdc5c6045231d875 in term 1.
I20260928 20:24:54.000947 32640 raft_consensus.cc:2454] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c456a93ad7bf4c76bdc5c6045231d875 in term 1.
I20260928 20:24:54.001245 32299 leader_election.cc:302] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f43d62f60d6443bb7e4a47d028df299, c456a93ad7bf4c76bdc5c6045231d875; no voters:
I20260928 20:24:54.001403 32727 raft_consensus.cc:2781] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:54.001721 32727 raft_consensus.cc:686] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 1 LEADER]: Becoming Leader. State: Replica: c456a93ad7bf4c76bdc5c6045231d875, State: Running, Role: LEADER
I20260928 20:24:54.002161 32727 consensus_queue.cc:237] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } }
I20260928 20:24:54.003470 31447 catalog_manager.cc:5495] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 reported cstate change: term changed from 0 to 1, leader changed from <none> to c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "c456a93ad7bf4c76bdc5c6045231d875" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } health_report { overall_health: UNKNOWN } } }
W20260928 20:24:54.030220 32415 tablet.cc:2369] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:54.039386 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:54.036533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.24.186.193:38787)
0928 20:24:54.036592 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:54.036594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:54.036603 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:54.036665 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:54.036666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:54.036692 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:54.036758 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:54.036947 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.037599 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.038341 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.038701 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.038758 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:54.038972 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:54.039080 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:54.039164 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:54.039299 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:24:54.039325 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:24:54.043051 32502 raft_consensus.cc:1261] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Refusing update from remote peer c456a93ad7bf4c76bdc5c6045231d875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:54.043118 32640 raft_consensus.cc:1261] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Refusing update from remote peer c456a93ad7bf4c76bdc5c6045231d875: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:54.043479 32729 consensus_queue.cc:1035] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:54.043629 32727 consensus_queue.cc:1035] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:54.048005 32733 mvcc.cc:205] Tried to move back new op lower bound from 7334408577196883968 to 7334408577034002432. Current Snapshot: MvccSnapshot[applied={T|T < 7334408577196883968}]
I20260928 20:24:54.048509 32734 mvcc.cc:205] Tried to move back new op lower bound from 7334408577196883968 to 7334408577034002432. Current Snapshot: MvccSnapshot[applied={T|T < 7334408577196883968 or (T in {7334408577196883968})}]
I20260928 20:24:54.049253 31446 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:54.049348 31446 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:54.051259 31446 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37cf5c1738184d959f42e8f4d3ac8ec4: Sending DeleteTablet for 3 replicas of tablet a9ff12535c9b4129bae439fbf8e9d7fb
I20260928 20:24:54.051546 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:54 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38634
I20260928 20:24:54.051610 32482 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:54 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57256
I20260928 20:24:54.051920 32346 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ff12535c9b4129bae439fbf8e9d7fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:54 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35214
I20260928 20:24:54.052176 32744 tablet_replica.cc:330] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: stopping tablet replica
I20260928 20:24:54.052278 32744 raft_consensus.cc:2229] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:54.052387 32744 raft_consensus.cc:2258] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:54.052758 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31415
I20260928 20:24:54.052803 32744 ts_tablet_manager.cc:1905] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:54.054957 32744 ts_tablet_manager.cc:1918] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:54.055039 32744 log.cc:1196] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/a9ff12535c9b4129bae439fbf8e9d7fb
I20260928 20:24:54.055307 32744 ts_tablet_manager.cc:1939] T a9ff12535c9b4129bae439fbf8e9d7fb P c456a93ad7bf4c76bdc5c6045231d875: Deleting consensus metadata
I20260928 20:24:54.059676 32742 tablet_replica.cc:330] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: stopping tablet replica
I20260928 20:24:54.059845 32742 raft_consensus.cc:2229] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:54.059984 32742 raft_consensus.cc:2258] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:54.060177 32743 tablet_replica.cc:330] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: stopping tablet replica
I20260928 20:24:54.060281 32743 raft_consensus.cc:2229] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:54.060384 32743 raft_consensus.cc:2258] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:54.061025 32743 ts_tablet_manager.cc:1905] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:54.061281 32742 ts_tablet_manager.cc:1905] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:54.062708 32743 ts_tablet_manager.cc:1918] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:54.063050 32743 log.cc:1196] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/a9ff12535c9b4129bae439fbf8e9d7fb
I20260928 20:24:54.063218 32742 ts_tablet_manager.cc:1918] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:54.063344 32742 log.cc:1196] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/a9ff12535c9b4129bae439fbf8e9d7fb
I20260928 20:24:54.063702 32743 ts_tablet_manager.cc:1939] T a9ff12535c9b4129bae439fbf8e9d7fb P 7f43d62f60d6443bb7e4a47d028df299: Deleting consensus metadata
I20260928 20:24:54.063859 32742 ts_tablet_manager.cc:1939] T a9ff12535c9b4129bae439fbf8e9d7fb P d76b21af060c44dc82dd9ce32f61981e: Deleting consensus metadata
W20260928 20:24:54.064254 32576 connection.cc:441] server connection from 127.0.0.1:38634 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38634 (request call id 1) could send its response
W20260928 20:24:54.064388 32438 connection.cc:441] server connection from 127.0.0.1:57256 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57256 (request call id 1) could send its response
I20260928 20:24:54.072381 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:54.063696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.24.186.253:38315)
0928 20:24:54.063790 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:54.063793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:54.064414 (+ 621us) server_negotiation.cc:407] Connection header received
0928 20:24:54.064545 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:54.064546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:54.064562 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:54.064614 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:54.065210 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.065946 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.067100 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.067362 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.067941 (+ 579us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:54.068695 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:54.068697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:54.068697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:54.068704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:54.070769 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:54.071250 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:54.071251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:54.071251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:54.071317 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:54.071535 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:54.071536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:54.071536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:54.071669 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:54.071780 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:54.072179 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 20:24:54.072277 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:24:54.072626 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:54.064084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.24.186.252:37251)
0928 20:24:54.064126 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:54.064128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:54.064547 (+ 419us) server_negotiation.cc:407] Connection header received
0928 20:24:54.064657 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:54.064658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:54.064671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:54.064712 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:54.065451 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.066130 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.066877 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:54.067131 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:54.067197 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:54.068759 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:54.068760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:54.068760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:54.068767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:54.070769 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:54.071325 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:54.071326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:54.071327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:54.071384 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:54.071564 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:54.071565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:54.071566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:54.071712 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:54.071926 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:54.072417 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 20:24:54.072521 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:55.049126 32752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.048559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:37823)
0928 20:24:55.048851 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.049024 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":179,"threads_started":1}
I20260928 20:24:55.049243 32753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.048883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:54741)
0928 20:24:55.049046 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.049156 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
W20260928 20:24:55.049510 32687 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
W20260928 20:24:55.049507 32549 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.052325 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.052110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:35293)
0928 20:24:55.052213 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.052271 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
W20260928 20:24:55.052562 32413 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33525 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.551879 32756 raft_consensus.cc:482] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37cf5c1738184d959f42e8f4d3ac8ec4)
I20260928 20:24:55.552011 32755 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 37cf5c1738184d959f42e8f4d3ac8ec4)
I20260928 20:24:55.552007 32756 raft_consensus.cc:504] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } }
I20260928 20:24:55.552182 32755 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } }
I20260928 20:24:55.552460 32755 leader_election.cc:288] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525), f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253:38315)
I20260928 20:24:55.552803 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38860)
0928 20:24:55.552610 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.552659 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":42,"threads_started":1}
I20260928 20:24:55.552829 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38858)
0928 20:24:55.552731 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.552781 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":46,"threads_started":1}
W20260928 20:24:55.552976 32215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.553189 32756 leader_election.cc:288] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525), ca4c087324bc44eea2de6ae1e956029e (127.24.186.252:37251)
W20260928 20:24:55.553246 32029 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.553622 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.553534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38870)
0928 20:24:55.553572 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.553595 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 20:24:55.553722 32029 leader_election.cc:334] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525): Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.553959 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.553858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38900)
0928 20:24:55.553896 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.553922 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 20:24:55.554028 32215 leader_election.cc:334] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525): Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.556102 32761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.24.186.253:38315)
0928 20:24:55.552936 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.552938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.552948 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:55.553027 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.553028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.553040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.553133 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.553284 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.553992 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.555194 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.555722 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.555826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.555835 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.556018 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.556052 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":48,"threads_started":1}
I20260928 20:24:55.556151 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:38315 (local address 127.0.0.1:51066)
0928 20:24:55.552873 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.552884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.552941 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.553133 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.553135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.553138 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.553202 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.553204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.554089 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.554090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.555114 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.555116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.555867 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.555868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.555939 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.555944 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.556014 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.556060 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260928 20:24:55.556380 32055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4c087324bc44eea2de6ae1e956029e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" is_pre_election: true
I20260928 20:24:55.556485 32055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4c087324bc44eea2de6ae1e956029e in term 1.
I20260928 20:24:55.556656 32218 leader_election.cc:302] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca4c087324bc44eea2de6ae1e956029e, f87c6b879e7c4a8ba3e71527cd60dbce; no voters: 37cf5c1738184d959f42e8f4d3ac8ec4
I20260928 20:24:55.556775 32755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 20:24:55.556830 32755 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 37cf5c1738184d959f42e8f4d3ac8ec4)
I20260928 20:24:55.556862 32755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:24:55.556938 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:37251 (local address 127.0.0.1:53798)
0928 20:24:55.552796 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.552806 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.553000 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.553321 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.553321 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.553323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.553355 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.553356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.554790 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.554791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.555851 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.555853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.556661 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.556662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.556758 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.556762 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.556825 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.556872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":41,"threads_started":1}
I20260928 20:24:55.557075 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.552863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.24.186.252:37251)
0928 20:24:55.553009 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.553011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.553018 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:55.553145 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.553146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.553156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.553208 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.553958 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.554722 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.555909 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.556556 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.556662 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.556671 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.556874 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.556934 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
I20260928 20:24:55.557245 32241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca4c087324bc44eea2de6ae1e956029e" is_pre_election: true
I20260928 20:24:55.557709 32755 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } }
I20260928 20:24:55.557855 32755 leader_election.cc:288] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 election: Requested vote from peers 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525), f87c6b879e7c4a8ba3e71527cd60dbce (127.24.186.253:38315)
I20260928 20:24:55.557863 32241 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f87c6b879e7c4a8ba3e71527cd60dbce in current term 2: Already voted for candidate ca4c087324bc44eea2de6ae1e956029e in this term.
I20260928 20:24:55.558072 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.557932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38916)
0928 20:24:55.557969 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.558051 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 20:24:55.558068 32029 leader_election.cc:302] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f87c6b879e7c4a8ba3e71527cd60dbce; no voters: 37cf5c1738184d959f42e8f4d3ac8ec4, ca4c087324bc44eea2de6ae1e956029e
I20260928 20:24:55.558063 32055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca4c087324bc44eea2de6ae1e956029e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce"
I20260928 20:24:55.558154 32055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:24:55.558351 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.558263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38932)
0928 20:24:55.558297 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.558329 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 20:24:55.558436 32215 leader_election.cc:334] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525): Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
I20260928 20:24:55.558895 32055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4c087324bc44eea2de6ae1e956029e in term 2.
I20260928 20:24:55.558974 32756 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:24:55.559059 32218 leader_election.cc:302] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ca4c087324bc44eea2de6ae1e956029e, f87c6b879e7c4a8ba3e71527cd60dbce; no voters: 37cf5c1738184d959f42e8f4d3ac8ec4
I20260928 20:24:55.559193 32755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:24:55.559341 32755 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [term 2 LEADER]: Becoming Leader. State: Replica: ca4c087324bc44eea2de6ae1e956029e, State: Running, Role: LEADER
I20260928 20:24:55.559419 32755 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } }
I20260928 20:24:55.560019 32765 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4c087324bc44eea2de6ae1e956029e. Latest consensus state: current_term: 2 leader_uuid: "ca4c087324bc44eea2de6ae1e956029e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:55.560211 32765 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:55.560508 32767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:55.561228 32767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:55.561443 32767 catalog_manager.cc:1259] Loaded cluster ID: 5280696460894460b2ce82cc9a75e73e
I20260928 20:24:55.561491 32767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:24:55.561671 32767 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:55.561766 32767 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Loaded TSK: 0
I20260928 20:24:55.561909 32767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:24:55.609097 32231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:55.610674 32231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:55.611784 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.611658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38942)
0928 20:24:55.611719 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.611759 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:24:55.611778 32055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [term 2 FOLLOWER]: Refusing update from remote peer ca4c087324bc44eea2de6ae1e956029e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 20:24:55.612076 32765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 20:24:55.612233 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.612118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.0.0.1:38958)
0928 20:24:55.612165 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.612189 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:24:55.612299 32215 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e -> Peer 37cf5c1738184d959f42e8f4d3ac8ec4 (127.24.186.254:33525): Couldn't send request to peer 37cf5c1738184d959f42e8f4d3ac8ec4. Status: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 20:24:55.612752 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4c087324bc44eea2de6ae1e956029e. Latest consensus state: current_term: 2 leader_uuid: "ca4c087324bc44eea2de6ae1e956029e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:55.612865 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:55.613289 301 mvcc.cc:205] Tried to move back new op lower bound from 7334408583623606272 to 7334408583412002816. Current Snapshot: MvccSnapshot[applied={T|T < 7334408583623606272}]
I20260928 20:24:55.613283 32755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ca4c087324bc44eea2de6ae1e956029e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:55.613368 32755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:55.613447 32756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ca4c087324bc44eea2de6ae1e956029e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "37cf5c1738184d959f42e8f4d3ac8ec4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33525 } } peers { permanent_uuid: "f87c6b879e7c4a8ba3e71527cd60dbce" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 38315 } attrs { promote: false } } peers { permanent_uuid: "ca4c087324bc44eea2de6ae1e956029e" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 37251 } attrs { promote: false } } }
I20260928 20:24:55.613544 32756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c6b879e7c4a8ba3e71527cd60dbce [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:24:55.618669 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45335 (local address 127.0.0.1:38640)
0928 20:24:55.615429 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.615437 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.615473 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.615636 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.615637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.615639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.615679 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.615681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616589 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.616590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.617679 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.617680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.618390 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.618391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.618454 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.618458 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.618528 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.618586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260928 20:24:55.618669 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.24.186.195:45335)
0928 20:24:55.615478 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.615480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.615505 (+ 25us) server_negotiation.cc:407] Connection header received
0928 20:24:55.615528 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.615529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.615539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.615591 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.615764 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616512 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.617813 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.618299 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.618392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.618402 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.618559 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.618600 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":52,"threads_started":1}
I20260928 20:24:55.618952 32620 tablet_service.cc:1467] Processing CreateTablet for tablet 48070c742a1e46e1a95a5b58c7590117 (DEFAULT_TABLE table=test-table [id=f6811303a1fe4ea781e40820b1305f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:55.619076 32620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48070c742a1e46e1a95a5b58c7590117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:55.619009 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38787 (local address 127.0.0.1:35238)
0928 20:24:55.615275 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.615288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.615334 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.615505 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.615506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.615509 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.615562 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.615564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616415 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.616416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.617557 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.617560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.618599 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.618600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.618669 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.618675 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.618708 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.618752 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260928 20:24:55.619750 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:42867 (local address 127.0.0.1:57268)
0928 20:24:55.615303 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.615313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.615374 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.615540 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.615541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.615542 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.615571 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.615572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616554 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.616555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.618012 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.618018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.619519 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.619520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.619609 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.619613 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.619644 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.619689 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":114}
I20260928 20:24:55.619740 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.24.186.193:38787)
0928 20:24:55.615320 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.615322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.615332 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:55.615396 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.615398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.615410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.615638 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.615674 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616336 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.617738 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.618371 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.619556 (+ 1185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.619573 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.619595 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.619666 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
I20260928 20:24:55.620009 32346 tablet_service.cc:1467] Processing CreateTablet for tablet 48070c742a1e46e1a95a5b58c7590117 (DEFAULT_TABLE table=test-table [id=f6811303a1fe4ea781e40820b1305f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:55.620061 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.615115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.24.186.194:42867)
0928 20:24:55.615260 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.615262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.615362 (+ 100us) server_negotiation.cc:407] Connection header received
0928 20:24:55.615433 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.615434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.615447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.615538 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.615626 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.616452 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.618192 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.619390 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.619884 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.619895 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.619920 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.619985 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 20:24:55.620132 32346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48070c742a1e46e1a95a5b58c7590117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:55.620518 309 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Bootstrap starting.
I20260928 20:24:55.620616 32482 tablet_service.cc:1467] Processing CreateTablet for tablet 48070c742a1e46e1a95a5b58c7590117 (DEFAULT_TABLE table=test-table [id=f6811303a1fe4ea781e40820b1305f02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:55.620728 32482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48070c742a1e46e1a95a5b58c7590117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:55.620898 309 tablet_bootstrap.cc:654] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:55.621546 309 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: No bootstrap required, opened a new log
I20260928 20:24:55.621606 309 ts_tablet_manager.cc:1397] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:55.621759 309 raft_consensus.cc:348] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.621840 309 raft_consensus.cc:374] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:55.621866 309 raft_consensus.cc:729] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76b21af060c44dc82dd9ce32f61981e, State: Initialized, Role: FOLLOWER
I20260928 20:24:55.621906 309 consensus_queue.cc:260] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.622087 309 ts_tablet_manager.cc:1428] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:55.622247 313 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Bootstrap starting.
I20260928 20:24:55.622614 311 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: Bootstrap starting.
I20260928 20:24:55.622646 313 tablet_bootstrap.cc:654] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:55.622786 32686 heartbeater.cc:503] Master 127.24.186.252:37251 was elected leader, sending a full tablet report...
I20260928 20:24:55.623160 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.622388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:36171)
0928 20:24:55.622998 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.623040 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":75,"threads_started":1}
I20260928 20:24:55.623237 313 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: No bootstrap required, opened a new log
I20260928 20:24:55.623262 311 tablet_bootstrap.cc:654] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:55.623281 313 ts_tablet_manager.cc:1397] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:55.623391 313 raft_consensus.cc:348] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.623436 313 raft_consensus.cc:374] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:55.623449 313 raft_consensus.cc:729] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f43d62f60d6443bb7e4a47d028df299, State: Initialized, Role: FOLLOWER
I20260928 20:24:55.623514 313 consensus_queue.cc:260] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.623940 313 ts_tablet_manager.cc:1428] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:55.624123 311 tablet_bootstrap.cc:492] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: No bootstrap required, opened a new log
I20260928 20:24:55.624191 311 ts_tablet_manager.cc:1397] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:55.624375 311 raft_consensus.cc:348] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.624588 311 raft_consensus.cc:374] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:55.624656 311 raft_consensus.cc:729] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c456a93ad7bf4c76bdc5c6045231d875, State: Initialized, Role: FOLLOWER
I20260928 20:24:55.624722 311 consensus_queue.cc:260] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.624806 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.624172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:44289)
0928 20:24:55.624737 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.624778 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542}
I20260928 20:24:55.624907 311 ts_tablet_manager.cc:1428] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:55.625073 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.624576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:60399)
0928 20:24:55.624988 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.625026 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":100,"threads_started":1}
I20260928 20:24:55.625279 32548 heartbeater.cc:503] Master 127.24.186.252:37251 was elected leader, sending a full tablet report...
I20260928 20:24:55.625464 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.625378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:47029)
0928 20:24:55.625413 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.625439 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 20:24:55.625515 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.625228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:39499)
0928 20:24:55.625440 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.625478 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
I20260928 20:24:55.625937 320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.625778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:40911)
0928 20:24:55.625886 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.625912 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260928 20:24:55.626194 32412 heartbeater.cc:503] Master 127.24.186.252:37251 was elected leader, sending a full tablet report...
I20260928 20:24:55.754813 312 raft_consensus.cc:482] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:55.754916 312 raft_consensus.cc:504] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.755250 312 leader_election.cc:288] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867), c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:55.758579 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.755510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:45395 (local address 127.24.186.193:38787)
0928 20:24:55.755597 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.755599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.755610 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:24:55.755643 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.755645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.755658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.755712 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.756007 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.756542 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.757614 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.758215 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.758289 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.758301 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.758488 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.758521 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:24:55.758600 314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.755450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38787 (local address 127.24.186.195:45395)
0928 20:24:55.755494 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.755506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.755563 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.755750 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.755751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.755755 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.755897 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.755900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.756650 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.756651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.757507 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.757510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.758322 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.758324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.758398 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.758402 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.758475 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.758520 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 20:24:55.759159 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48070c742a1e46e1a95a5b58c7590117" candidate_uuid: "d76b21af060c44dc82dd9ce32f61981e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c456a93ad7bf4c76bdc5c6045231d875" is_pre_election: true
I20260928 20:24:55.759344 32366 raft_consensus.cc:2454] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d76b21af060c44dc82dd9ce32f61981e in term 0.
I20260928 20:24:55.759500 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.755456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:42867 (local address 127.24.186.195:35977)
0928 20:24:55.755836 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.755845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.755891 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.756012 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:55.756012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:55.756014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:55.756062 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.756064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.756995 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.756997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.758167 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:55.758170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.759258 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.759259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:55.759326 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.759330 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:55.759385 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:24:55.759427 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":162,"thread_start_us":127,"threads_started":1}
I20260928 20:24:55.759496 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.755516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:35977 (local address 127.24.186.194:42867)
0928 20:24:55.755596 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.755599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.755893 (+ 294us) server_negotiation.cc:407] Connection header received
0928 20:24:55.755930 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.755931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.755941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.755996 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:55.756163 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.756686 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.758319 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.759149 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.759253 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.759265 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.759391 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.759433 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:24:55.759605 32574 leader_election.cc:302] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c456a93ad7bf4c76bdc5c6045231d875, d76b21af060c44dc82dd9ce32f61981e; no voters:
I20260928 20:24:55.759742 312 raft_consensus.cc:2781] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:55.759707 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48070c742a1e46e1a95a5b58c7590117" candidate_uuid: "d76b21af060c44dc82dd9ce32f61981e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f43d62f60d6443bb7e4a47d028df299" is_pre_election: true
I20260928 20:24:55.759783 312 raft_consensus.cc:482] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:55.759824 312 raft_consensus.cc:3037] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:55.759814 32502 raft_consensus.cc:2454] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d76b21af060c44dc82dd9ce32f61981e in term 0.
I20260928 20:24:55.760569 312 raft_consensus.cc:504] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.760721 312 leader_election.cc:288] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [CANDIDATE]: Term 1 election: Requested vote from peers 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867), c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787)
I20260928 20:24:55.760882 32502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48070c742a1e46e1a95a5b58c7590117" candidate_uuid: "d76b21af060c44dc82dd9ce32f61981e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f43d62f60d6443bb7e4a47d028df299"
I20260928 20:24:55.760928 32366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48070c742a1e46e1a95a5b58c7590117" candidate_uuid: "d76b21af060c44dc82dd9ce32f61981e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c456a93ad7bf4c76bdc5c6045231d875"
I20260928 20:24:55.760965 32502 raft_consensus.cc:3037] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:55.761009 32366 raft_consensus.cc:3037] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:55.761709 32502 raft_consensus.cc:2454] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d76b21af060c44dc82dd9ce32f61981e in term 1.
I20260928 20:24:55.761786 32366 raft_consensus.cc:2454] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d76b21af060c44dc82dd9ce32f61981e in term 1.
I20260928 20:24:55.761957 32573 leader_election.cc:302] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f43d62f60d6443bb7e4a47d028df299, d76b21af060c44dc82dd9ce32f61981e; no voters:
I20260928 20:24:55.762068 312 raft_consensus.cc:2781] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:55.762207 312 raft_consensus.cc:686] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 1 LEADER]: Becoming Leader. State: Replica: d76b21af060c44dc82dd9ce32f61981e, State: Running, Role: LEADER
I20260928 20:24:55.762265 312 consensus_queue.cc:237] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } }
I20260928 20:24:55.762538 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.762443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:40823)
0928 20:24:55.762478 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.762508 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 20:24:55.762970 32231 catalog_manager.cc:5495] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e reported cstate change: term changed from 0 to 1, leader changed from <none> to d76b21af060c44dc82dd9ce32f61981e (127.24.186.195). New cstate: current_term: 1 leader_uuid: "d76b21af060c44dc82dd9ce32f61981e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d76b21af060c44dc82dd9ce32f61981e" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45335 } health_report { overall_health: HEALTHY } } }
I20260928 20:24:55.802772 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.800415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.24.186.195:45335)
0928 20:24:55.800457 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:55.800460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:55.800468 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:55.800517 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:55.800518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:55.800536 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:55.800616 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:55.800729 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.801211 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.801972 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:55.802173 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:55.802254 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:55.802423 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:55.802522 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:55.802602 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:55.802655 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 20:24:55.802679 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:55.803966 32502 raft_consensus.cc:1261] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Refusing update from remote peer d76b21af060c44dc82dd9ce32f61981e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:55.804262 32366 raft_consensus.cc:1261] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Refusing update from remote peer d76b21af060c44dc82dd9ce32f61981e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:55.804272 323 consensus_queue.cc:1035] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f43d62f60d6443bb7e4a47d028df299" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:55.804502 323 consensus_queue.cc:1035] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c456a93ad7bf4c76bdc5c6045231d875" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:55.805080 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.804941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:52271)
0928 20:24:55.805010 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.805048 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 20:24:55.806001 321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.805874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:43883)
0928 20:24:55.805917 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.805967 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 20:24:55.806330 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.806142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:34339)
0928 20:24:55.806271 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.806299 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 20:24:55.806394 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.806311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.194:48901)
0928 20:24:55.806345 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.806366 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 20:24:55.807770 322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.807686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.195:35441)
0928 20:24:55.807726 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.807746 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 20:24:55.808116 321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:55.808030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33525 (local address 127.24.186.193:36041)
0928 20:24:55.808069 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.808090 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 20:24:55.808382 329 mvcc.cc:205] Tried to move back new op lower bound from 7334408584410685440 to 7334408584242774016. Current Snapshot: MvccSnapshot[applied={T|T < 7334408584410685440 or (T in {7334408584410685440})}]
I20260928 20:24:55.808964 32230 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:55.809048 32230 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:55.810264 32230 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca4c087324bc44eea2de6ae1e956029e: Sending DeleteTablet for 3 replicas of tablet 48070c742a1e46e1a95a5b58c7590117
I20260928 20:24:55.810540 32346 tablet_service.cc:1514] Processing DeleteTablet for tablet 48070c742a1e46e1a95a5b58c7590117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:55 UTC) from {username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:35238
I20260928 20:24:55.810719 335 tablet_replica.cc:330] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875: stopping tablet replica
I20260928 20:24:55.810810 32482 tablet_service.cc:1514] Processing DeleteTablet for tablet 48070c742a1e46e1a95a5b58c7590117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:55 UTC) from {username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:57268
I20260928 20:24:55.810848 32620 tablet_service.cc:1514] Processing DeleteTablet for tablet 48070c742a1e46e1a95a5b58c7590117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:55 UTC) from {username='oryx', principal='oryx/127.24.186.252@KRBTEST.COM'} at 127.0.0.1:38640
I20260928 20:24:55.810847 335 raft_consensus.cc:2229] T 48070c742a1e46e1a95a5b58c7590117 P c456a93ad7bf4c76bdc5c6045231d875 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:55.810937 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 32283
I20260928 20:24:55.810988 336 tablet_replica.cc:330] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: stopping tablet replica
I20260928 20:24:55.811038 336 raft_consensus.cc:2229] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:55.811079 336 raft_consensus.cc:2258] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:55.811944 336 ts_tablet_manager.cc:1905] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:55.813578 336 ts_tablet_manager.cc:1918] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:55.813681 336 log.cc:1196] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/48070c742a1e46e1a95a5b58c7590117
I20260928 20:24:55.813930 336 ts_tablet_manager.cc:1939] T 48070c742a1e46e1a95a5b58c7590117 P 7f43d62f60d6443bb7e4a47d028df299: Deleting consensus metadata
I20260928 20:24:55.814321 32215 catalog_manager.cc:4841] TS 7f43d62f60d6443bb7e4a47d028df299 (127.24.186.194:42867): tablet 48070c742a1e46e1a95a5b58c7590117 (table test-table [id=f6811303a1fe4ea781e40820b1305f02]) successfully deleted
I20260928 20:24:55.815945 337 tablet_replica.cc:330] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: stopping tablet replica
I20260928 20:24:55.816030 337 raft_consensus.cc:2229] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:55.816133 337 raft_consensus.cc:2258] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:55.816422 337 ts_tablet_manager.cc:1905] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:55.817783 337 ts_tablet_manager.cc:1918] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:55.817862 337 log.cc:1196] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/48070c742a1e46e1a95a5b58c7590117
I20260928 20:24:55.818109 337 ts_tablet_manager.cc:1939] T 48070c742a1e46e1a95a5b58c7590117 P d76b21af060c44dc82dd9ce32f61981e: Deleting consensus metadata
I20260928 20:24:55.818480 32215 catalog_manager.cc:4841] TS d76b21af060c44dc82dd9ce32f61981e (127.24.186.195:45335): tablet 48070c742a1e46e1a95a5b58c7590117 (table test-table [id=f6811303a1fe4ea781e40820b1305f02]) successfully deleted
W20260928 20:24:55.819621 32759 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:55.819238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38787 (local address 127.0.0.1:35248)
0928 20:24:55.819349 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:55.819361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:55.819398 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:55.819571 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:38787: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 20:24:55.819911 32216 catalog_manager.cc:4584] TS c456a93ad7bf4c76bdc5c6045231d875 (127.24.186.193:38787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48070c742a1e46e1a95a5b58c7590117: Network error: Client connection negotiation failed: client connection to 127.24.186.193:38787: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:24:55.820086 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 32419
I20260928 20:24:55.826021 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 32555
I20260928 20:24:55.831739 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 31911
I20260928 20:24:55.837150 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 32099
2026-09-28T20:24:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4946 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:55 dist-test-slave-k477 krb5kdc[340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:55 dist-test-slave-k477 krb5kdc[340](info): set up 2 sockets
Sep 28 20:24:55 dist-test-slave-k477 krb5kdc[340](info): commencing operation
krb5kdc: starting...
W20260928 20:24:56.909247 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:24:56 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:24:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab.
Entry for principal oryx/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab.
I20260928 20:24:57.040216 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34891
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab
--principal=oryx/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:34891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:57.123838 353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:57.124130 353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:57.124172 353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:57.125388 353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:24:57.125463 353 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 20:24:57.125485 353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:57.125514 353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:57.125541 353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:24:57.125571 353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:24:57.127170 353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:34891
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.254.keytab
--principal=oryx/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:57.127457 353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:57.127693 353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:57.132871 359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.132993 353 server_base.cc:1034] running on GCE node
W20260928 20:24:57.133033 361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:57.133087 358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.133466 353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:57.133800 353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:57.135005 353 hybrid_clock.cc:648] HybridClock initialized: now 1790627097135026 us; error 68 us; skew 500 ppm
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:57.137713 353 init.cc:377] Logged in from keytab as oryx/127.24.186.254@KRBTEST.COM (short username oryx)
I20260928 20:24:57.138161 353 webserver.cc:460] Webserver started at http://127.24.186.254:33193/ using document root <none> and password file <none>
I20260928 20:24:57.138410 353 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:57.138478 353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:57.138631 353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:57.140107 353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "d38fb2f80d62464782704486519a4de1"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "6647869836c059266063f462493a3de0"
server_key_iv: "d9745fcec678475ee6fd2b04d800d276"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.140487 353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "d38fb2f80d62464782704486519a4de1"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "6647869836c059266063f462493a3de0"
server_key_iv: "d9745fcec678475ee6fd2b04d800d276"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.142961 353 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:24:57.144016 368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.144352 353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:57.144474 353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "d38fb2f80d62464782704486519a4de1"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "6647869836c059266063f462493a3de0"
server_key_iv: "d9745fcec678475ee6fd2b04d800d276"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.144569 353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:57.191277 353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:57.193871 353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:57.194043 353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:57.198755 353 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:34891
I20260928 20:24:57.198840 430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:34891 every 8 connection(s)
I20260928 20:24:57.199170 353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:24:57.199898 431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:57.202144 431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Bootstrap starting.
I20260928 20:24:57.202785 431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:57.203168 431 log.cc:824] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:57.204007 431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: No bootstrap required, opened a new log
I20260928 20:24:57.206082 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 353
I20260928 20:24:57.206226 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:24:57.206421 25323 external_mini_cluster.cc:1428] Setting key 4c6dacb21cea730c4a49de48631017ca
I20260928 20:24:57.206965 431 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } }
I20260928 20:24:57.207167 431 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:57.207216 431 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38fb2f80d62464782704486519a4de1, State: Initialized, Role: FOLLOWER
I20260928 20:24:57.207345 431 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } }
I20260928 20:24:57.207429 431 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:24:57.207470 431 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:24:57.207523 431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:57.208338 431 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } }
I20260928 20:24:57.208441 431 leader_election.cc:302] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38fb2f80d62464782704486519a4de1; no voters:
I20260928 20:24:57.208593 431 leader_election.cc:288] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:24:57.208631 436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:57.208778 436 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [term 1 LEADER]: Becoming Leader. State: Replica: d38fb2f80d62464782704486519a4de1, State: Running, Role: LEADER
I20260928 20:24:57.208861 431 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:24:57.208859 436 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } }
I20260928 20:24:57.209532 436 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38fb2f80d62464782704486519a4de1. Latest consensus state: current_term: 1 leader_uuid: "d38fb2f80d62464782704486519a4de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } } }
I20260928 20:24:57.209653 436 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:24:57.209628 438 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38fb2f80d62464782704486519a4de1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38fb2f80d62464782704486519a4de1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34891 } } }
I20260928 20:24:57.209712 438 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1 [sys.catalog]: This master's current role is: LEADER
W20260928 20:24:57.210886 451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:24:57.210952 451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:24:57.211019 445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:24:57.211205 445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:24:57.212793 445 catalog_manager.cc:1347] Generated new cluster ID: e549b76c10c04cc8a60739392dc32b6b
I20260928 20:24:57.212841 445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:57.220000 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.206974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.24.186.254:34891)
0928 20:24:57.207149 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.207160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.207195 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:24:57.207331 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.207339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.207401 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.207546 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.208361 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.209801 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.210973 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.211589 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.211706 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.214726 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.214731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.214731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.214745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.217536 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.218114 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.218122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.218123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.218398 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.218792 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.218793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.218793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.219197 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.219379 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.219452 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.219732 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260928 20:24:57.230634 445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:24:57.231191 445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:24:57.237478 445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Generated new TSK 0
I20260928 20:24:57.237720 445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
I20260928 20:24:57.274360 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34891
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:57.366763 457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:57.366997 457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:57.367024 457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:57.368535 457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:57.368599 457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:57.368631 457 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:57.368654 457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:24:57.371065 457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:57.371285 457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:57.371476 457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:57.376616 463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:57.376621 462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.376748 457 server_base.cc:1034] running on GCE node
W20260928 20:24:57.376612 465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.377022 457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:57.377265 457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:57.378458 457 hybrid_clock.cc:648] HybridClock initialized: now 1790627097378423 us; error 48 us; skew 500 ppm
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:57.381207 457 init.cc:377] Logged in from keytab as oryx/127.24.186.193@KRBTEST.COM (short username oryx)
I20260928 20:24:57.381762 457 webserver.cc:460] Webserver started at http://127.24.186.193:41569/ using document root <none> and password file <none>
I20260928 20:24:57.382025 457 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:57.382120 457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:57.382277 457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:57.383386 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "929de691888343a8878814cd4fe519cd"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "f2cc795a7b3ab8781fd5e41a2016b849"
server_key_iv: "92db88f50ee4453b54fb87fb0ecc75ca"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.383764 457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "929de691888343a8878814cd4fe519cd"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "f2cc795a7b3ab8781fd5e41a2016b849"
server_key_iv: "92db88f50ee4453b54fb87fb0ecc75ca"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.385301 457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:57.386132 472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.386312 457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:57.386421 457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "929de691888343a8878814cd4fe519cd"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "f2cc795a7b3ab8781fd5e41a2016b849"
server_key_iv: "92db88f50ee4453b54fb87fb0ecc75ca"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.386503 457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:57.398674 457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:57.401207 457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:57.401378 457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:57.401932 457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:57.401995 457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.402051 457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:57.402088 457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.408996 457 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:33033
I20260928 20:24:57.409150 587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:33033 every 8 connection(s)
I20260928 20:24:57.409442 457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:57.411389 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 457
I20260928 20:24:57.411489 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:24:57.411693 25323 external_mini_cluster.cc:1428] Setting key d8e653705110925235ffce300a3c9263
I20260928 20:24:57.413863 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.402576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47523 (local address 127.24.186.254:34891)
0928 20:24:57.402643 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.402647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.402848 (+ 201us) server_negotiation.cc:407] Connection header received
0928 20:24:57.403381 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.403383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.403399 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.403476 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.404589 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.405451 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.407082 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.407459 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.407546 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.410470 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.410472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.410472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.410481 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.412294 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.412829 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.412831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.412831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.412895 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.413166 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.413167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.413167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.413292 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.413373 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.413680 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.413770 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:57.414079 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.402538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.193:47523)
0928 20:24:57.402714 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.402756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.403249 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.403598 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.403605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.404153 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.404455 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.404477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.405579 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.405580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.406994 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.406998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.407553 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.407554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.407693 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.408456 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.408464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.410365 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.412388 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.412396 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.412398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.412756 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.412973 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.412973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.412974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.413094 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.413364 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.413370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.413536 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.413658 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.413832 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260928 20:24:57.414565 588 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34891
I20260928 20:24:57.414662 588 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:57.414865 588 heartbeater.cc:511] Master 127.24.186.254:34891 requested a full tablet report, sending...
I20260928 20:24:57.415436 384 ts_manager.cc:194] Registered new tserver with Master: 929de691888343a8878814cd4fe519cd (127.24.186.193:33033)
I20260928 20:24:57.416383 384 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:47523
WARNING: no policy specified for oryx/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.194@KRBTEST.COM" created.
I20260928 20:24:57.427554 595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.415756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.193:47535)
0928 20:24:57.419977 (+ 4221us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.419991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.420081 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.420229 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.420230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.421149 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.421271 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.421278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.422398 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.422401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.423066 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.423069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.423485 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.423486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.423537 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.424090 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.424097 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.424483 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.426413 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.426414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.426415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.426599 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.426861 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.426862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.426862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.426929 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.427212 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.427212 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.427267 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.427365 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.427478 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4181,"thread_start_us":100,"threads_started":1}
I20260928 20:24:57.427662 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.415817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47535 (local address 127.24.186.254:34891)
0928 20:24:57.415862 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.415865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.420094 (+ 4229us) server_negotiation.cc:407] Connection header received
0928 20:24:57.420150 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.420151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.420162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.420221 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.421479 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.422259 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.423197 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.423407 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.423477 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.424562 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.424563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.424563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.424570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.426304 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.426719 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.426721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.426721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.426787 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.426998 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.426999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.426999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.427133 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.427222 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.427427 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.427526 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
I20260928 20:24:57.443451 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34891
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:57.527096 597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:57.527320 597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:57.527346 597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:57.528668 597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:57.528715 597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:57.528741 597 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:57.528760 597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:24:57.530316 597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:57.530530 597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:57.530726 597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:57.535912 603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:57.535912 602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.536054 597 server_base.cc:1034] running on GCE node
W20260928 20:24:57.535912 605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.536370 597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:57.536599 597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:57.537740 597 hybrid_clock.cc:648] HybridClock initialized: now 1790627097537722 us; error 43 us; skew 500 ppm
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:57.540426 597 init.cc:377] Logged in from keytab as oryx/127.24.186.194@KRBTEST.COM (short username oryx)
I20260928 20:24:57.540901 597 webserver.cc:460] Webserver started at http://127.24.186.194:34617/ using document root <none> and password file <none>
I20260928 20:24:57.541096 597 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:57.541137 597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:57.541226 597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:57.542320 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "1d1294edb4629e7a9944a724447f9941"
server_key_iv: "2e9752df47c77f5fb46717fe5e9f4827"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.542678 597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "1d1294edb4629e7a9944a724447f9941"
server_key_iv: "2e9752df47c77f5fb46717fe5e9f4827"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.544090 597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:24:57.545717 612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.546083 597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:57.546229 597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "1d1294edb4629e7a9944a724447f9941"
server_key_iv: "2e9752df47c77f5fb46717fe5e9f4827"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.546322 597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:57.560845 597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:57.563329 597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:57.563453 597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:57.564016 597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:57.564082 597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.564127 597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:57.564155 597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.571661 597 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:46241
I20260928 20:24:57.571736 727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:46241 every 8 connection(s)
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:57.572193 597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:24:57.575198 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.564424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48825 (local address 127.24.186.254:34891)
0928 20:24:57.564492 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.564495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.564696 (+ 201us) server_negotiation.cc:407] Connection header received
0928 20:24:57.565225 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.565227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.565239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.565336 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.566304 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.567312 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.568958 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.569310 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.569404 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.572540 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.572542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.572542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.572552 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.573973 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.574397 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.574398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.574398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.574445 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.574662 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.574663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.574663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.574770 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.574820 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.575070 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.575132 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:24:57.575400 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.564407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.194:48825)
0928 20:24:57.564552 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.564595 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.565122 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.565391 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.565395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.565797 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.566119 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.566126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.567428 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.567430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.568843 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.568847 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.569425 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.569426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.569594 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.570268 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.570279 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.572397 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.574058 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.574064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.574066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.574326 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.574501 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.574502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.574502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.574600 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.574825 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.574829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.574950 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.575043 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.575172 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260928 20:24:57.575970 728 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34891
I20260928 20:24:57.576057 728 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:57.576259 728 heartbeater.cc:511] Master 127.24.186.254:34891 requested a full tablet report, sending...
I20260928 20:24:57.576656 385 ts_manager.cc:194] Registered new tserver with Master: 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241)
I20260928 20:24:57.577167 385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:48825
I20260928 20:24:57.579581 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 597
I20260928 20:24:57.579708 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:24:57.579974 25323 external_mini_cluster.cc:1428] Setting key 3738bec79e48b450b36e8d0e6e55b36b
I20260928 20:24:57.583577 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.577100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.194:54689)
0928 20:24:57.577276 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.577293 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.577370 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.577600 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.577601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.577836 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.577907 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.577910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.578813 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.578814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.579371 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.579373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.579941 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.579943 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.580022 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.580537 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.580542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.581013 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.582511 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.582512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.582512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.582696 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.582906 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.582907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.582907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.582966 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.583311 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.583312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.583354 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.583417 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.583505 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260928 20:24:57.583689 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.577165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:54689 (local address 127.24.186.254:34891)
0928 20:24:57.577234 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.577236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.577499 (+ 263us) server_negotiation.cc:407] Connection header received
0928 20:24:57.577519 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.577520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.577529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.577575 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.578106 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.578675 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.579486 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.579853 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.579929 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.581094 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.581095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.581095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.581100 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.582421 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.582771 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.582772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.582773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.582830 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.583036 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.583037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.583037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.583250 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.583320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.583459 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.583546 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
I20260928 20:24:57.607764 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34891
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:24:57.693934 737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:24:57.694196 737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:24:57.694238 737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:24:57.695497 737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:24:57.695571 737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:24:57.695608 737 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 20:24:57.695643 737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:24:57.697353 737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.24.186.254:34891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:24:57.697634 737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:24:57.697952 737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:24:57.703985 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:57.704213 743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:24:57.704711 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:24:57.705207 737 server_base.cc:1034] running on GCE node
I20260928 20:24:57.705443 737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:24:57.705695 737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:24:57.706900 737 hybrid_clock.cc:648] HybridClock initialized: now 1790627097706878 us; error 37 us; skew 500 ppm
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:24:57.716089 737 init.cc:377] Logged in from keytab as oryx/127.24.186.195@KRBTEST.COM (short username oryx)
I20260928 20:24:57.716578 737 webserver.cc:460] Webserver started at http://127.24.186.195:43761/ using document root <none> and password file <none>
I20260928 20:24:57.716848 737 fs_manager.cc:359] Metadata directory not provided
I20260928 20:24:57.716897 737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:24:57.717027 737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:24:57.718113 737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "cb5e1f7c28b87bbd647d2dfedc511501"
server_key_iv: "f78ed53a2a50363324cba518a1a9d242"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.718482 737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "cb5e1f7c28b87bbd647d2dfedc511501"
server_key_iv: "f78ed53a2a50363324cba518a1a9d242"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.719949 737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:24:57.720813 752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.721030 737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:24:57.721128 737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
format_stamp: "Formatted at 2026-09-28 20:24:57 on dist-test-slave-k477"
server_key: "cb5e1f7c28b87bbd647d2dfedc511501"
server_key_iv: "f78ed53a2a50363324cba518a1a9d242"
server_key_version: "encryptionkey@0"
I20260928 20:24:57.721205 737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:24:57.732074 737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:24:57.734575 737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:24:57.734746 737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:24:57.735306 737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:24:57.735343 737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:57.735390 737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:24:57.735422 737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:57.743551 737 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45623
I20260928 20:24:57.743635 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45623 every 8 connection(s)
I20260928 20:24:57.744016 737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:24:57.747725 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.736015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:51715 (local address 127.24.186.254:34891)
0928 20:24:57.736081 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.736084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.736315 (+ 231us) server_negotiation.cc:407] Connection header received
0928 20:24:57.737192 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.737194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.737214 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.737287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.738389 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.739180 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.741001 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.741358 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.741438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.744355 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.744357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.744357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.744366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.745598 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.746128 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.746129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.746130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.746189 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.746817 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.746818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.746818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.746955 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.747020 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.747578 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.747658 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:57.748239 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.735992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.195:51715)
0928 20:24:57.736173 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.736217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.736807 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.737430 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.737436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.737898 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.738255 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.738260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.739315 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.739316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.740879 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.740883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.741453 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.741454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.741610 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.742279 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.742295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.744195 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.745705 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.745711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.745713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.745977 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.746424 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.746426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.746427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.746649 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.747045 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.747052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.747282 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.747502 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.747961 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260928 20:24:57.748886 868 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34891
I20260928 20:24:57.748996 868 heartbeater.cc:464] Registering TS with master...
I20260928 20:24:57.749303 868 heartbeater.cc:511] Master 127.24.186.254:34891 requested a full tablet report, sending...
I20260928 20:24:57.749711 385 ts_manager.cc:194] Registered new tserver with Master: 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:57.750182 385 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:51715
I20260928 20:24:57.754047 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 737
I20260928 20:24:57.754190 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:24:57.754391 25323 external_mini_cluster.cc:1428] Setting key e1743556029251974e5707d4f67b3f2b
I20260928 20:24:57.755374 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:24:57.756578 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.750513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:37987 (local address 127.24.186.254:34891)
0928 20:24:57.750565 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.750567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.750713 (+ 146us) server_negotiation.cc:407] Connection header received
0928 20:24:57.750775 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.750776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.750787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.750843 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.751225 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.751713 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.752439 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.752657 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.752745 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.753725 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.753726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.753727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.753734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.755422 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.755761 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.755762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.755763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.755843 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.756034 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.756035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.756035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.756161 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.756241 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.756407 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.756484 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:24:57.756592 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.750463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.24.186.195:37987)
0928 20:24:57.750647 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.750657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.750715 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.750848 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.750849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.751041 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:57.751134 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.751137 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.751834 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.751835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.752352 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.752354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.752747 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.752748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.752812 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.753294 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:57.753298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:57.753618 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:57.755495 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.755497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.755497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.755688 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.755905 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:57.755905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:57.755906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:57.755969 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:57.756232 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:57.756233 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:57.756298 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.756381 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.756488 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":7,"thread_start_us":88,"threads_started":1}
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:24:57.779259 435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:57.773315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.24.186.254:34891)
0928 20:24:57.773382 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.773384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.773392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:57.773417 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.773418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.773427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.773486 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.774033 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.774531 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.775612 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.775895 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.775984 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.779196 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39780: BlockingRecv error: recv got EOF from 127.0.0.1:39780 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 20:24:57 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:57.788089 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.780332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.24.186.254:34891)
0928 20:24:57.780382 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.780384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.780476 (+ 92us) server_negotiation.cc:407] Connection header received
0928 20:24:57.780521 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.780522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.780531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.780580 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:57.781166 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.781636 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.782318 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.782499 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.782562 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.785153 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:57.785156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:57.785156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:57.785163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:57.786848 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.787304 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.787306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.787306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.787374 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:57.787556 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:57.787556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:57.787557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:57.787678 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:57.787761 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.787944 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.788019 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:24:57.789801 385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:57.791399 385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:57.798094 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.794275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.24.186.195:45623)
0928 20:24:57.794460 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.794464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.794471 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:24:57.794546 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.794548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.794565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.794640 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:57.794840 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.795545 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797119 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797645 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797770 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.797834 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.797952 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.797988 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260928 20:24:57.798208 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.793734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46241 (local address 127.0.0.1:54840)
0928 20:24:57.793991 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.794033 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.794145 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.794563 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.794566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.794568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:57.794641 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.794643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.796023 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.796023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.797101 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.797103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797952 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.798024 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.798027 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.798056 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.798106 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":53,"threads_started":1}
I20260928 20:24:57.798298 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.794043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.24.186.193:33033)
0928 20:24:57.794681 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.794685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.794694 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:57.794730 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.794732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.794747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.794852 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:57.794966 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.796023 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797064 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797785 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797879 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.797937 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.798151 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.798195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":74,"threads_started":1}
I20260928 20:24:57.798409 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.794057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.24.186.194:46241)
0928 20:24:57.794387 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:57.794392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:57.794402 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:57.794442 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:57.794444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:57.794459 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:57.794557 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:57.794729 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.795923 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797178 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797857 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.798187 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.798242 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:57.798268 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 20:24:57.798306 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":106,"threads_started":1}
I20260928 20:24:57.798089 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.794080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45623 (local address 127.0.0.1:37256)
0928 20:24:57.794249 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.794259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.794305 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.794701 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.794702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.794704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:57.794763 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.794764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.795636 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.795637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.797014 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.797016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797754 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.797840 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.797854 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.797925 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.797987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
I20260928 20:24:57.798503 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:57.793920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.0.0.1:41522)
0928 20:24:57.794206 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:57.794216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:57.794269 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:57.794855 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:57.794856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:57.794858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:57.794891 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.794892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.796086 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.796087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.796980 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:57.796982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:57.797966 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:57.797967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:57.798020 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:57.798024 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:57.798072 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 20:24:57.798110 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260928 20:24:57.799233 662 tablet_service.cc:1467] Processing CreateTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 (DEFAULT_TABLE table=test-table [id=07025426a6b94c45ab335acc1cfc38b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:57.799616 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965fa6c1765c4019b4efb95b36bc0627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:57.799700 522 tablet_service.cc:1467] Processing CreateTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 (DEFAULT_TABLE table=test-table [id=07025426a6b94c45ab335acc1cfc38b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:57.799700 802 tablet_service.cc:1467] Processing CreateTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 (DEFAULT_TABLE table=test-table [id=07025426a6b94c45ab335acc1cfc38b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:57.800472 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965fa6c1765c4019b4efb95b36bc0627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:57.800493 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965fa6c1765c4019b4efb95b36bc0627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:57.803028 896 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Bootstrap starting.
I20260928 20:24:57.803668 896 tablet_bootstrap.cc:654] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:57.804008 896 log.cc:824] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:57.804363 897 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Bootstrap starting.
I20260928 20:24:57.804515 899 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Bootstrap starting.
I20260928 20:24:57.805079 899 tablet_bootstrap.cc:654] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:57.805076 897 tablet_bootstrap.cc:654] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:57.805163 896 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: No bootstrap required, opened a new log
I20260928 20:24:57.805246 896 ts_tablet_manager.cc:1397] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:57.805429 897 log.cc:824] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:57.805781 899 log.cc:824] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Log is configured to *not* fsync() on all Append() calls
I20260928 20:24:57.806568 897 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: No bootstrap required, opened a new log
I20260928 20:24:57.806648 897 ts_tablet_manager.cc:1397] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:57.806830 899 tablet_bootstrap.cc:492] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: No bootstrap required, opened a new log
I20260928 20:24:57.806916 899 ts_tablet_manager.cc:1397] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:24:57.807574 896 raft_consensus.cc:348] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.807771 896 raft_consensus.cc:374] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:57.807843 896 raft_consensus.cc:729] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Initialized, Role: FOLLOWER
I20260928 20:24:57.807989 896 consensus_queue.cc:260] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.808264 896 ts_tablet_manager.cc:1428] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:57.808386 728 heartbeater.cc:503] Master 127.24.186.254:34891 was elected leader, sending a full tablet report...
I20260928 20:24:57.808863 897 raft_consensus.cc:348] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.808861 899 raft_consensus.cc:348] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.809027 899 raft_consensus.cc:374] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:57.809027 897 raft_consensus.cc:374] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:57.809074 899 raft_consensus.cc:729] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929de691888343a8878814cd4fe519cd, State: Initialized, Role: FOLLOWER
I20260928 20:24:57.809106 897 raft_consensus.cc:729] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Initialized, Role: FOLLOWER
I20260928 20:24:57.809181 899 consensus_queue.cc:260] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.809243 897 consensus_queue.cc:260] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:57.809470 588 heartbeater.cc:503] Master 127.24.186.254:34891 was elected leader, sending a full tablet report...
I20260928 20:24:57.809499 897 ts_tablet_manager.cc:1428] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:24:57.809633 868 heartbeater.cc:503] Master 127.24.186.254:34891 was elected leader, sending a full tablet report...
I20260928 20:24:57.809479 899 ts_tablet_manager.cc:1428] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 20:24:57.822834 729 tablet.cc:2369] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:57.910373 589 tablet.cc:2369] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:24:57.994664 869 tablet.cc:2369] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:24:58.000334 904 raft_consensus.cc:482] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:58.000450 904 raft_consensus.cc:504] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.000778 904 leader_election.cc:288] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241), 929de691888343a8878814cd4fe519cd (127.24.186.193:33033)
I20260928 20:24:58.004747 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.001134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:51103 (local address 127.24.186.193:33033)
0928 20:24:58.001215 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.001217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.001355 (+ 138us) server_negotiation.cc:407] Connection header received
0928 20:24:58.001383 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.001384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.001391 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.001428 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.001621 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.002331 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.003772 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.004354 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.004432 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.004440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.004653 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.004691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:24:58.004747 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.001072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:33103 (local address 127.24.186.194:46241)
0928 20:24:58.001120 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.001122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.001157 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:24:58.001206 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.001207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.001217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.001279 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.001477 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.002249 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.003772 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.004355 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.004432 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.004440 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.004633 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.004672 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:24:58.004768 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.001072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.24.186.195:51103)
0928 20:24:58.001228 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.001245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.001320 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.001451 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.001452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.001453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.001522 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.001523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.002410 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.002411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.003674 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.003677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.004446 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.004447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.004542 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.004546 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.004618 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.004661 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":35,"thread_start_us":89,"threads_started":1}
I20260928 20:24:58.004778 761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.000951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46241 (local address 127.24.186.195:33103)
0928 20:24:58.001022 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.001036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.001131 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.001299 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.001301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.001303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.001380 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.001381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.002346 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.002347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.003674 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.003677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.004446 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.004447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.004545 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.004547 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.004611 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.004661 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":51}
I20260928 20:24:58.005273 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965fa6c1765c4019b4efb95b36bc0627" candidate_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd" is_pre_election: true
I20260928 20:24:58.005321 682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965fa6c1765c4019b4efb95b36bc0627" candidate_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" is_pre_election: true
I20260928 20:24:58.005442 542 raft_consensus.cc:2454] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9619eea451ef4070b4df8ba7ea11fc3d in term 0.
I20260928 20:24:58.005465 682 raft_consensus.cc:2454] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9619eea451ef4070b4df8ba7ea11fc3d in term 0.
I20260928 20:24:58.005676 755 leader_election.cc:302] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 929de691888343a8878814cd4fe519cd, 9619eea451ef4070b4df8ba7ea11fc3d; no voters:
I20260928 20:24:58.005800 904 raft_consensus.cc:2781] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:58.005854 904 raft_consensus.cc:482] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:58.005878 904 raft_consensus.cc:3037] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.006837 904 raft_consensus.cc:504] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.006980 904 leader_election.cc:288] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [CANDIDATE]: Term 1 election: Requested vote from peers 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241), 929de691888343a8878814cd4fe519cd (127.24.186.193:33033)
I20260928 20:24:58.007162 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965fa6c1765c4019b4efb95b36bc0627" candidate_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd"
I20260928 20:24:58.007162 682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965fa6c1765c4019b4efb95b36bc0627" candidate_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f"
I20260928 20:24:58.007242 542 raft_consensus.cc:3037] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.007242 682 raft_consensus.cc:3037] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.008111 682 raft_consensus.cc:2454] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9619eea451ef4070b4df8ba7ea11fc3d in term 1.
I20260928 20:24:58.008104 542 raft_consensus.cc:2454] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9619eea451ef4070b4df8ba7ea11fc3d in term 1.
I20260928 20:24:58.008266 755 leader_election.cc:302] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 929de691888343a8878814cd4fe519cd, 9619eea451ef4070b4df8ba7ea11fc3d; no voters:
I20260928 20:24:58.008355 904 raft_consensus.cc:2781] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:58.008514 904 raft_consensus.cc:686] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 LEADER]: Becoming Leader. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Running, Role: LEADER
I20260928 20:24:58.008666 904 consensus_queue.cc:237] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.009595 385 catalog_manager.cc:5495] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195). New cstate: current_term: 1 leader_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } health_report { overall_health: HEALTHY } } }
I20260928 20:24:58.032492 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.030149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.24.186.195:45623)
0928 20:24:58.030215 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.030217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.030226 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:58.030273 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.030274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.030290 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.030342 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.030492 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.030971 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.031710 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.031973 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.032032 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.032176 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.032272 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.032331 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.032366 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.032384 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:24:58.034235 682 raft_consensus.cc:1261] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Refusing update from remote peer 9619eea451ef4070b4df8ba7ea11fc3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.034283 542 raft_consensus.cc:1261] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Refusing update from remote peer 9619eea451ef4070b4df8ba7ea11fc3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.034628 906 consensus_queue.cc:1035] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.034760 904 consensus_queue.cc:1035] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.037400 911 mvcc.cc:205] Tried to move back new op lower bound from 7334408593545498624 to 7334408593444155392. Current Snapshot: MvccSnapshot[applied={T|T < 7334408593545498624}]
I20260928 20:24:58.039338 384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.039451 384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.040239 384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Sending DeleteTablet for 3 replicas of tablet 965fa6c1765c4019b4efb95b36bc0627
I20260928 20:24:58.040544 662 tablet_service.cc:1514] Processing DeleteTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54840
I20260928 20:24:58.040791 918 tablet_replica.cc:330] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: stopping tablet replica
I20260928 20:24:58.040779 522 tablet_service.cc:1514] Processing DeleteTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:41522
I20260928 20:24:58.040800 384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:58.040947 384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:58.040990 802 tablet_service.cc:1514] Processing DeleteTablet for tablet 965fa6c1765c4019b4efb95b36bc0627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37256
I20260928 20:24:58.041172 918 raft_consensus.cc:2229] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.041361 920 tablet_replica.cc:330] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: stopping tablet replica
I20260928 20:24:58.041471 920 raft_consensus.cc:2229] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:58.041599 920 raft_consensus.cc:2258] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.041726 919 tablet_replica.cc:330] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: stopping tablet replica
I20260928 20:24:58.041823 919 raft_consensus.cc:2229] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.041923 919 raft_consensus.cc:2258] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.042630 662 tablet_service.cc:1467] Processing CreateTablet for tablet 6af2ab9c75374e48a853a69394830c7c (DEFAULT_TABLE table=test-table [id=9f0ec59806b64011a38ab63645a60a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.042740 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af2ab9c75374e48a853a69394830c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.042927 918 raft_consensus.cc:2258] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.043149 919 ts_tablet_manager.cc:1905] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.043680 896 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Bootstrap starting.
I20260928 20:24:58.043845 522 tablet_service.cc:1467] Processing CreateTablet for tablet 6af2ab9c75374e48a853a69394830c7c (DEFAULT_TABLE table=test-table [id=9f0ec59806b64011a38ab63645a60a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.043956 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af2ab9c75374e48a853a69394830c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.044128 896 tablet_bootstrap.cc:654] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.044445 918 ts_tablet_manager.cc:1905] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.045017 899 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Bootstrap starting.
I20260928 20:24:58.042454 920 ts_tablet_manager.cc:1905] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.043740 802 tablet_service.cc:1467] Processing CreateTablet for tablet 6af2ab9c75374e48a853a69394830c7c (DEFAULT_TABLE table=test-table [id=9f0ec59806b64011a38ab63645a60a26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.045322 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af2ab9c75374e48a853a69394830c7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.045390 899 tablet_bootstrap.cc:654] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.045961 899 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: No bootstrap required, opened a new log
I20260928 20:24:58.046017 899 ts_tablet_manager.cc:1397] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.046175 899 raft_consensus.cc:348] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.046264 899 raft_consensus.cc:374] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.046293 899 raft_consensus.cc:729] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929de691888343a8878814cd4fe519cd, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.046334 899 consensus_queue.cc:260] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.046433 899 ts_tablet_manager.cc:1428] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:58.047909 897 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Bootstrap starting.
I20260928 20:24:58.048444 897 tablet_bootstrap.cc:654] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.048785 920 ts_tablet_manager.cc:1918] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.048861 920 log.cc:1196] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/965fa6c1765c4019b4efb95b36bc0627
I20260928 20:24:58.049105 919 ts_tablet_manager.cc:1918] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.049222 897 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: No bootstrap required, opened a new log
I20260928 20:24:58.049306 897 ts_tablet_manager.cc:1397] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.049281 919 log.cc:1196] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/965fa6c1765c4019b4efb95b36bc0627
I20260928 20:24:58.049451 918 ts_tablet_manager.cc:1918] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.049444 897 raft_consensus.cc:348] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.049490 897 raft_consensus.cc:374] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.049504 897 raft_consensus.cc:729] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.049535 897 consensus_queue.cc:260] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.049628 897 ts_tablet_manager.cc:1428] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.049216 920 ts_tablet_manager.cc:1939] T 965fa6c1765c4019b4efb95b36bc0627 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting consensus metadata
I20260928 20:24:58.050084 371 catalog_manager.cc:4841] TS 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623): tablet 965fa6c1765c4019b4efb95b36bc0627 (table test-table [id=07025426a6b94c45ab335acc1cfc38b4]) successfully deleted
I20260928 20:24:58.050335 918 log.cc:1196] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/965fa6c1765c4019b4efb95b36bc0627
I20260928 20:24:58.050379 896 tablet_bootstrap.cc:492] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: No bootstrap required, opened a new log
I20260928 20:24:58.050875 896 ts_tablet_manager.cc:1397] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 20:24:58.051043 896 raft_consensus.cc:348] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.051108 896 raft_consensus.cc:374] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.051137 896 raft_consensus.cc:729] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.051186 896 consensus_queue.cc:260] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.051271 896 ts_tablet_manager.cc:1428] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.051733 918 ts_tablet_manager.cc:1939] T 965fa6c1765c4019b4efb95b36bc0627 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting consensus metadata
I20260928 20:24:58.052057 919 ts_tablet_manager.cc:1939] T 965fa6c1765c4019b4efb95b36bc0627 P 929de691888343a8878814cd4fe519cd: Deleting consensus metadata
I20260928 20:24:58.052479 371 catalog_manager.cc:4841] TS 929de691888343a8878814cd4fe519cd (127.24.186.193:33033): tablet 965fa6c1765c4019b4efb95b36bc0627 (table test-table [id=07025426a6b94c45ab335acc1cfc38b4]) successfully deleted
I20260928 20:24:58.052713 371 catalog_manager.cc:4841] TS 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241): tablet 965fa6c1765c4019b4efb95b36bc0627 (table test-table [id=07025426a6b94c45ab335acc1cfc38b4]) successfully deleted
I20260928 20:24:58.125628 902 raft_consensus.cc:482] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:58.125727 902 raft_consensus.cc:504] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.126062 902 leader_election.cc:288] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.129627 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.126330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:41929 (local address 127.24.186.193:33033)
0928 20:24:58.126395 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.126398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.126482 (+ 84us) server_negotiation.cc:407] Connection header received
0928 20:24:58.126602 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.126603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.126614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.126645 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.126871 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.127617 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.128882 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.129233 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.129320 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.129329 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.129500 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.129548 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:24:58.129635 924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.126240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.24.186.194:41929)
0928 20:24:58.126413 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.126425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.126513 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.126718 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.126719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.126721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.126800 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.126802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.127705 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.127706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.128793 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.128795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.129321 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.129322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.129394 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.129401 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.129484 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.129530 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":75,"threads_started":1}
I20260928 20:24:58.129726 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.126491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:44115 (local address 127.24.186.195:45623)
0928 20:24:58.126530 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.126532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.126567 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:24:58.126652 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.126653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.126664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.126752 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.126903 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.127371 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.128767 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.129358 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.129438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.129446 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.129609 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.129655 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:24:58.130074 925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.126423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45623 (local address 127.24.186.194:44115)
0928 20:24:58.126517 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.126526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.126587 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.126765 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.126766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.126767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.126812 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.126813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.127462 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.127463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.128670 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.128673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.129447 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.129448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.129517 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.129519 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.129926 (+ 407us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.129979 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260928 20:24:58.130199 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af2ab9c75374e48a853a69394830c7c" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd" is_pre_election: true
I20260928 20:24:58.130296 542 raft_consensus.cc:2454] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.130338 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af2ab9c75374e48a853a69394830c7c" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" is_pre_election: true
I20260928 20:24:58.130463 822 raft_consensus.cc:2454] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.130483 615 leader_election.cc:302] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.130592 902 raft_consensus.cc:2781] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:58.130628 902 raft_consensus.cc:482] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:58.130654 902 raft_consensus.cc:3037] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.131322 902 raft_consensus.cc:504] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.131453 902 leader_election.cc:288] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Requested vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.131633 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af2ab9c75374e48a853a69394830c7c" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd"
I20260928 20:24:58.131652 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af2ab9c75374e48a853a69394830c7c" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
I20260928 20:24:58.131723 542 raft_consensus.cc:3037] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.131737 822 raft_consensus.cc:3037] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.132375 822 raft_consensus.cc:2454] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.132532 615 leader_election.cc:302] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 9619eea451ef4070b4df8ba7ea11fc3d; no voters:
I20260928 20:24:58.132615 902 raft_consensus.cc:2781] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:58.132611 542 raft_consensus.cc:2454] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.132717 902 raft_consensus.cc:686] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Becoming Leader. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Running, Role: LEADER
I20260928 20:24:58.132784 902 consensus_queue.cc:237] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.133515 385 catalog_manager.cc:5495] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194). New cstate: current_term: 1 leader_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } health_report { overall_health: UNKNOWN } } }
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627097, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.254@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:58.163283 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.156662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.24.186.254:34891)
0928 20:24:58.156722 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.156724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.156871 (+ 147us) server_negotiation.cc:407] Connection header received
0928 20:24:58.156939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.156940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.156950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.157004 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:58.157473 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.157952 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.158638 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.158822 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.158898 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.160963 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:58.160965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:58.160965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:58.160971 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:58.162364 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:58.162670 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:58.162671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:58.162671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:58.162724 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:58.162888 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:58.162888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:58.162889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:58.162986 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:58.163036 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.163150 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.163218 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:24:58.163444 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.156597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34891 (local address 127.0.0.1:39802)
0928 20:24:58.156789 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.156798 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.156883 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.157007 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.157008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.157327 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:24:58.157371 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.157373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.158035 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.158036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.158551 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.158553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.158895 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.158896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.158946 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.159447 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:24:58.159451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:24:58.160878 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:24:58.162435 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:58.162440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:58.162442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:58.162610 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:58.162780 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:24:58.162781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:24:58.162781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:24:58.162836 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:24:58.163041 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:24:58.163046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:24:58.163085 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.163141 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.163282 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":53,"thread_start_us":75,"threads_started":1}
I20260928 20:24:58.165336 385 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:39802:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 20:24:58.165575 385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:58.167023 522 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3189b886d445db568364b686577a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0870b19c75574319a0b868f1925bced6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:58.167016 662 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3189b886d445db568364b686577a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0870b19c75574319a0b868f1925bced6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:58.167030 802 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3189b886d445db568364b686577a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0870b19c75574319a0b868f1925bced6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 20:24:58.167167 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3189b886d445db568364b686577a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.167168 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3189b886d445db568364b686577a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.167168 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3189b886d445db568364b686577a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.168296 897 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: Bootstrap starting.
I20260928 20:24:58.168597 897 tablet_bootstrap.cc:654] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.168614 896 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Bootstrap starting.
I20260928 20:24:58.168627 899 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: Bootstrap starting.
I20260928 20:24:58.169036 896 tablet_bootstrap.cc:654] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.169041 899 tablet_bootstrap.cc:654] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.169703 896 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: No bootstrap required, opened a new log
I20260928 20:24:58.169752 896 ts_tablet_manager.cc:1397] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.169767 897 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: No bootstrap required, opened a new log
I20260928 20:24:58.169807 897 ts_tablet_manager.cc:1397] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:58.169833 899 tablet_bootstrap.cc:492] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: No bootstrap required, opened a new log
I20260928 20:24:58.169866 899 ts_tablet_manager.cc:1397] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.169901 896 raft_consensus.cc:348] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.169907 897 raft_consensus.cc:348] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.169966 897 raft_consensus.cc:374] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.169966 896 raft_consensus.cc:374] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.169989 897 raft_consensus.cc:729] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.169989 896 raft_consensus.cc:729] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.169967 899 raft_consensus.cc:348] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.170012 899 raft_consensus.cc:374] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.170028 899 raft_consensus.cc:729] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929de691888343a8878814cd4fe519cd, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.170039 896 consensus_queue.cc:260] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.170038 897 consensus_queue.cc:260] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.170109 899 consensus_queue.cc:260] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.170122 896 ts_tablet_manager.cc:1428] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.170122 897 ts_tablet_manager.cc:1428] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.170197 899 ts_tablet_manager.cc:1428] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.170264 903 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.170287 902 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.170358 902 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.170353 903 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.170454 906 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.170547 906 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.191380 903 raft_consensus.cc:482] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:58.191480 903 raft_consensus.cc:504] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.191773 903 leader_election.cc:288] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.194923 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.192337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:37795 (local address 127.24.186.195:45623)
0928 20:24:58.192400 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.192402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.192412 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:24:58.192463 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.192464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.192473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.192526 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.192684 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.193159 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194191 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.194549 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194646 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.194655 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.194812 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.194846 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:24:58.194941 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.192098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48491 (local address 127.24.186.194:46241)
0928 20:24:58.192175 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.192177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.192313 (+ 136us) server_negotiation.cc:407] Connection header received
0928 20:24:58.192345 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.192346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.192356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.192443 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:24:58.192635 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.193125 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194198 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.194555 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194655 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.194663 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.194834 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.194874 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 20:24:58.194955 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.192018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46241 (local address 127.24.186.193:48491)
0928 20:24:58.192194 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.192206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.192268 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.192448 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.192449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.192452 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.192532 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.192534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.193209 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.193210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.194102 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.194105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.194655 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.194750 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.194753 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.194816 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.194858 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":61,"thread_start_us":124,"threads_started":1}
I20260928 20:24:58.195216 682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3189b886d445db568364b686577a9" candidate_uuid: "929de691888343a8878814cd4fe519cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" is_pre_election: true
I20260928 20:24:58.195307 682 raft_consensus.cc:2454] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 929de691888343a8878814cd4fe519cd in term 0.
I20260928 20:24:58.194949 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.192255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45623 (local address 127.24.186.193:37795)
0928 20:24:58.192361 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.192370 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.192408 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.192535 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.192536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.192537 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:24:58.192615 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.192617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.193245 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.193246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.194099 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.194101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.194648 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.194649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.194727 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.194730 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.194804 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.194846 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":50,"thread_start_us":75,"threads_started":1}
I20260928 20:24:58.195484 475 leader_election.cc:302] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.195680 903 raft_consensus.cc:2781] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:58.195727 903 raft_consensus.cc:482] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:58.195760 903 raft_consensus.cc:3037] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.195734 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3189b886d445db568364b686577a9" candidate_uuid: "929de691888343a8878814cd4fe519cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" is_pre_election: true
I20260928 20:24:58.195847 822 raft_consensus.cc:2454] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 929de691888343a8878814cd4fe519cd in term 0.
I20260928 20:24:58.196516 903 raft_consensus.cc:504] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.196656 903 leader_election.cc:288] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [CANDIDATE]: Term 1 election: Requested vote from peers 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.196812 682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3189b886d445db568364b686577a9" candidate_uuid: "929de691888343a8878814cd4fe519cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f"
I20260928 20:24:58.196883 682 raft_consensus.cc:3037] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.196841 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3189b886d445db568364b686577a9" candidate_uuid: "929de691888343a8878814cd4fe519cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
I20260928 20:24:58.196910 822 raft_consensus.cc:3037] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.197468 682 raft_consensus.cc:2454] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 929de691888343a8878814cd4fe519cd in term 1.
I20260928 20:24:58.197474 822 raft_consensus.cc:2454] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 929de691888343a8878814cd4fe519cd in term 1.
I20260928 20:24:58.197618 475 leader_election.cc:302] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.197695 903 raft_consensus.cc:2781] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:58.197834 903 raft_consensus.cc:686] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [term 1 LEADER]: Becoming Leader. State: Replica: 929de691888343a8878814cd4fe519cd, State: Running, Role: LEADER
I20260928 20:24:58.197918 903 consensus_queue.cc:237] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.198253 939 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: TxnStatusTablet state changed. Reason: New leader 929de691888343a8878814cd4fe519cd. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.198343 939 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:58.198503 941 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 20:24:58.198639 383 catalog_manager.cc:5495] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 929de691888343a8878814cd4fe519cd (127.24.186.193). New cstate: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:58.198606 941 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 20:24:58.209905 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.206058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.24.186.193:33033)
0928 20:24:58.206144 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.206146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.206155 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:58.206305 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.206306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.206326 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.206386 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.206578 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.207452 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.208891 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.209206 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.209300 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.209583 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.209702 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.209779 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.209804 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.209822 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 20:24:58.209944 933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.205998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.0.0.1:41528)
0928 20:24:58.206080 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.206093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.206143 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.206442 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.206443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.206444 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:58.206495 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.206497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.207889 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.207890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.208775 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.208779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.209391 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.209392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.209462 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.209481 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.209766 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.209769 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.209792 (+ 23us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.209877 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 20:24:58.210999 682 raft_consensus.cc:1261] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Refusing update from remote peer 929de691888343a8878814cd4fe519cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.211097 822 raft_consensus.cc:1261] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Refusing update from remote peer 929de691888343a8878814cd4fe519cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.211266 903 consensus_queue.cc:1035] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.211388 939 consensus_queue.cc:1035] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.212071 906 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: TxnStatusTablet state changed. Reason: New leader 929de691888343a8878814cd4fe519cd. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.212169 906 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.212123 902 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: TxnStatusTablet state changed. Reason: New leader 929de691888343a8878814cd4fe519cd. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.212232 902 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.212711 903 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.212793 903 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:58.213342 903 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.213410 903 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 929de691888343a8878814cd4fe519cd: This TxnStatusTablet replica's current role is: LEADER
I20260928 20:24:58.213724 909 mvcc.cc:205] Tried to move back new op lower bound from 7334408594270121984 to 7334408594219098112. Current Snapshot: MvccSnapshot[applied={T|T < 7334408594270121984}]
I20260928 20:24:58.213726 906 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.213845 906 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 9619eea451ef4070b4df8ba7ea11fc3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.214391 910 mvcc.cc:205] Tried to move back new op lower bound from 7334408594270121984 to 7334408594219098112. Current Snapshot: MvccSnapshot[applied={T|T < 7334408594270121984}]
I20260928 20:24:58.215045 902 tablet_replica.cc:439] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "929de691888343a8878814cd4fe519cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } } }
I20260928 20:24:58.215530 902 tablet_replica.cc:442] T 1dc3189b886d445db568364b686577a9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 20:24:58.217841 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.214379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.24.186.194:46241)
0928 20:24:58.214494 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.214496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.214504 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:24:58.214533 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.214533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.214549 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.214631 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.215030 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.216130 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.217078 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.217285 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.217362 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.217502 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.217620 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.217703 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.217745 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.217763 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 20:24:58.223205 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.220585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:59935 (local address 127.24.186.193:33033)
0928 20:24:58.220630 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.220632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.220736 (+ 104us) server_negotiation.cc:407] Connection header received
0928 20:24:58.220808 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.220809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.220819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.220876 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.221014 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.221739 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.222463 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.222775 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.222847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.222980 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.223013 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.223065 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.223108 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.223129 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:24:58.223213 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.220504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.24.186.194:59935)
0928 20:24:58.220659 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.220670 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.220735 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.220889 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.220890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.220891 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:58.220947 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.220948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.221817 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.221817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.222388 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.222389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.222849 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.222849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.222895 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.222907 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.223069 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.223071 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.223103 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.223134 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260928 20:24:58.225544 542 raft_consensus.cc:1261] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.225634 822 raft_consensus.cc:1261] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.225940 926 consensus_queue.cc:1035] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.226068 902 consensus_queue.cc:1035] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.233934 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.231099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47565 (local address 127.24.186.194:46241)
0928 20:24:58.231144 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.231146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.231286 (+ 140us) server_negotiation.cc:407] Connection header received
0928 20:24:58.231357 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.231358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.231368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.231406 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.231556 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.232086 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.233178 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.233403 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.233475 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.233676 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.233712 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.233762 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.233846 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.233864 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 20:24:58.233973 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.231072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46241 (local address 127.24.186.193:47565)
0928 20:24:58.231210 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.231222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:24:58.231300 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:24:58.231417 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:24:58.231418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:24:58.231420 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 20:24:58.231478 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.231480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.232180 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.232181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.233087 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:24:58.233089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.233484 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.233485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:24:58.233543 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.233564 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.233762 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.233764 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:24:58.233841 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 20:24:58.233882 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260928 20:24:58.243256 384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.243377 384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.244223 384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Sending DeleteTablet for 3 replicas of tablet 6af2ab9c75374e48a853a69394830c7c
I20260928 20:24:58.244534 522 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af2ab9c75374e48a853a69394830c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:41522
I20260928 20:24:58.244534 802 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af2ab9c75374e48a853a69394830c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37256
I20260928 20:24:58.244639 920 tablet_replica.cc:330] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: stopping tablet replica
I20260928 20:24:58.244640 919 tablet_replica.cc:330] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: stopping tablet replica
I20260928 20:24:58.244695 920 raft_consensus.cc:2229] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.244696 919 raft_consensus.cc:2229] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.244731 920 raft_consensus.cc:2258] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.244735 919 raft_consensus.cc:2258] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.244778 662 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af2ab9c75374e48a853a69394830c7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54840
I20260928 20:24:58.244863 918 tablet_replica.cc:330] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: stopping tablet replica
I20260928 20:24:58.244906 918 raft_consensus.cc:2229] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:58.244987 918 raft_consensus.cc:2258] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.245281 918 ts_tablet_manager.cc:1905] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.245334 920 ts_tablet_manager.cc:1905] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.247176 920 ts_tablet_manager.cc:1918] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:58.247232 920 log.cc:1196] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/6af2ab9c75374e48a853a69394830c7c
I20260928 20:24:58.247284 919 ts_tablet_manager.cc:1905] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.247445 920 ts_tablet_manager.cc:1939] T 6af2ab9c75374e48a853a69394830c7c P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting consensus metadata
I20260928 20:24:58.247881 371 catalog_manager.cc:4841] TS 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623): tablet 6af2ab9c75374e48a853a69394830c7c (table test-table [id=9f0ec59806b64011a38ab63645a60a26]) successfully deleted
I20260928 20:24:58.249051 919 ts_tablet_manager.cc:1918] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:58.249115 919 log.cc:1196] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/6af2ab9c75374e48a853a69394830c7c
I20260928 20:24:58.249354 919 ts_tablet_manager.cc:1939] T 6af2ab9c75374e48a853a69394830c7c P 929de691888343a8878814cd4fe519cd: Deleting consensus metadata
I20260928 20:24:58.249567 918 ts_tablet_manager.cc:1918] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:58.249634 918 log.cc:1196] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/6af2ab9c75374e48a853a69394830c7c
I20260928 20:24:58.249812 918 ts_tablet_manager.cc:1939] T 6af2ab9c75374e48a853a69394830c7c P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting consensus metadata
I20260928 20:24:58.249884 371 catalog_manager.cc:4841] TS 929de691888343a8878814cd4fe519cd (127.24.186.193:33033): tablet 6af2ab9c75374e48a853a69394830c7c (table test-table [id=9f0ec59806b64011a38ab63645a60a26]) successfully deleted
I20260928 20:24:58.250275 371 catalog_manager.cc:4841] TS 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241): tablet 6af2ab9c75374e48a853a69394830c7c (table test-table [id=9f0ec59806b64011a38ab63645a60a26]) successfully deleted
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:24:58.282567 435 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:58.275594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.24.186.254:34891)
0928 20:24:58.275662 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.275664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.275866 (+ 202us) server_negotiation.cc:407] Connection header received
0928 20:24:58.275912 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.275913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.275926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.275991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:58.276706 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.277538 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.278623 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.278938 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.279009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.282491 (+ 3482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39810: BlockingRecv error: recv got EOF from 127.0.0.1:39810 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM
I20260928 20:24:58.292441 435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.283364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.24.186.254:34891)
0928 20:24:58.283403 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.283405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.283635 (+ 230us) server_negotiation.cc:407] Connection header received
0928 20:24:58.283690 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.283691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.283703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.283765 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:24:58.284378 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.285138 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.286178 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.286435 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.286508 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.289307 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:24:58.289309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:24:58.289309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:24:58.289318 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:24:58.291065 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:58.291504 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:58.291505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:58.291506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:58.291569 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:24:58.291795 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:24:58.291796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:24:58.291796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:24:58.291982 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:24:58.292084 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.292256 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.292340 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:24:58.294461 384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:24:58.294641 384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:58.295969 662 tablet_service.cc:1467] Processing CreateTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 (DEFAULT_TABLE table=test-table [id=50c9739b24ed4ceaa914f461f099b535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.296125 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcccf77a7e884b8a8581ac46f72c80b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.297441 896 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Bootstrap starting.
I20260928 20:24:58.297477 802 tablet_service.cc:1467] Processing CreateTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 (DEFAULT_TABLE table=test-table [id=50c9739b24ed4ceaa914f461f099b535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.297592 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcccf77a7e884b8a8581ac46f72c80b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.297911 896 tablet_bootstrap.cc:654] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.298946 897 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Bootstrap starting.
I20260928 20:24:58.298981 522 tablet_service.cc:1467] Processing CreateTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 (DEFAULT_TABLE table=test-table [id=50c9739b24ed4ceaa914f461f099b535]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.299096 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcccf77a7e884b8a8581ac46f72c80b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.299291 897 tablet_bootstrap.cc:654] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.300477 899 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Bootstrap starting.
I20260928 20:24:58.300778 896 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: No bootstrap required, opened a new log
I20260928 20:24:58.300822 896 ts_tablet_manager.cc:1397] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:24:58.300839 899 tablet_bootstrap.cc:654] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.300952 896 raft_consensus.cc:348] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.301019 896 raft_consensus.cc:374] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.301039 896 raft_consensus.cc:729] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.301079 896 consensus_queue.cc:260] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.301157 896 ts_tablet_manager.cc:1428] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.301549 897 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: No bootstrap required, opened a new log
I20260928 20:24:58.301606 897 ts_tablet_manager.cc:1397] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:24:58.301702 897 raft_consensus.cc:348] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.301759 897 raft_consensus.cc:374] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.301779 897 raft_consensus.cc:729] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.301823 897 consensus_queue.cc:260] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.301914 897 ts_tablet_manager.cc:1428] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.302024 899 tablet_bootstrap.cc:492] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: No bootstrap required, opened a new log
I20260928 20:24:58.302093 899 ts_tablet_manager.cc:1397] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:58.302268 899 raft_consensus.cc:348] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.302346 899 raft_consensus.cc:374] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.302368 899 raft_consensus.cc:729] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929de691888343a8878814cd4fe519cd, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.302409 899 consensus_queue.cc:260] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.302496 899 ts_tablet_manager.cc:1428] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.582399 902 raft_consensus.cc:482] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:58.582501 902 raft_consensus.cc:504] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.582729 902 leader_election.cc:288] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.583007 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcccf77a7e884b8a8581ac46f72c80b9" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd" is_pre_election: true
I20260928 20:24:58.583102 542 raft_consensus.cc:2454] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.583035 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcccf77a7e884b8a8581ac46f72c80b9" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" is_pre_election: true
I20260928 20:24:58.583199 822 raft_consensus.cc:2454] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.583287 615 leader_election.cc:302] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.583389 902 raft_consensus.cc:2781] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:58.583465 902 raft_consensus.cc:482] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:58.583498 902 raft_consensus.cc:3037] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.584272 902 raft_consensus.cc:504] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.584438 902 leader_election.cc:288] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Requested vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.584625 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcccf77a7e884b8a8581ac46f72c80b9" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd"
I20260928 20:24:58.584631 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcccf77a7e884b8a8581ac46f72c80b9" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
I20260928 20:24:58.584703 542 raft_consensus.cc:3037] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.584704 822 raft_consensus.cc:3037] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.585472 822 raft_consensus.cc:2454] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.585472 542 raft_consensus.cc:2454] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.585673 615 leader_election.cc:302] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.585816 902 raft_consensus.cc:2781] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:58.585894 902 raft_consensus.cc:686] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Becoming Leader. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Running, Role: LEADER
I20260928 20:24:58.585954 902 consensus_queue.cc:237] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.586622 384 catalog_manager.cc:5495] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194). New cstate: current_term: 1 leader_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:58.659638 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.657021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.24.186.194:46241)
0928 20:24:58.657082 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:24:58.657084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:24:58.657093 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:24:58.657128 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:24:58.657128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:24:58.657137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:24:58.657193 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:24:58.657347 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.657911 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.658813 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:24:58.658997 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:24:58.659061 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:24:58.659278 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:24:58.659308 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:24:58.659353 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 20:24:58.659534 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 20:24:58.659559 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:24:58.661033 542 raft_consensus.cc:1261] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.661139 822 raft_consensus.cc:1261] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.661576 954 consensus_queue.cc:1035] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.661698 902 consensus_queue.cc:1035] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.664419 384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.664510 384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.665310 384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Sending DeleteTablet for 3 replicas of tablet bcccf77a7e884b8a8581ac46f72c80b9
I20260928 20:24:58.665583 522 tablet_service.cc:1514] Processing DeleteTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:41522
I20260928 20:24:58.665596 662 tablet_service.cc:1514] Processing DeleteTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54840
I20260928 20:24:58.665606 802 tablet_service.cc:1514] Processing DeleteTablet for tablet bcccf77a7e884b8a8581ac46f72c80b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37256
I20260928 20:24:58.665689 918 tablet_replica.cc:330] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: stopping tablet replica
I20260928 20:24:58.665710 920 tablet_replica.cc:330] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: stopping tablet replica
I20260928 20:24:58.665742 918 raft_consensus.cc:2229] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:58.665753 919 tablet_replica.cc:330] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: stopping tablet replica
I20260928 20:24:58.665774 920 raft_consensus.cc:2229] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.665817 920 raft_consensus.cc:2258] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.665814 919 raft_consensus.cc:2229] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.665820 918 raft_consensus.cc:2258] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.665860 919 raft_consensus.cc:2258] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.666177 918 ts_tablet_manager.cc:1905] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.666195 384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 20:24:58.666332 919 ts_tablet_manager.cc:1905] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 20:24:58.666371 384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:24:58.667629 662 tablet_service.cc:1467] Processing CreateTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 (DEFAULT_TABLE table=test-table [id=cce2b0b7115f44dd89bc32617a76ca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.667671 802 tablet_service.cc:1467] Processing CreateTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 (DEFAULT_TABLE table=test-table [id=cce2b0b7115f44dd89bc32617a76ca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.667742 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b9f637965b4d8ba6b5d8bfafc884c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.667834 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b9f637965b4d8ba6b5d8bfafc884c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.667923 918 ts_tablet_manager.cc:1918] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.667922 919 ts_tablet_manager.cc:1918] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.667973 918 log.cc:1196] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/bcccf77a7e884b8a8581ac46f72c80b9
I20260928 20:24:58.667975 919 log.cc:1196] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/bcccf77a7e884b8a8581ac46f72c80b9
I20260928 20:24:58.668192 919 ts_tablet_manager.cc:1939] T bcccf77a7e884b8a8581ac46f72c80b9 P 929de691888343a8878814cd4fe519cd: Deleting consensus metadata
I20260928 20:24:58.668452 918 ts_tablet_manager.cc:1939] T bcccf77a7e884b8a8581ac46f72c80b9 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting consensus metadata
I20260928 20:24:58.669163 371 catalog_manager.cc:4841] TS 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194:46241): tablet bcccf77a7e884b8a8581ac46f72c80b9 (table test-table [id=50c9739b24ed4ceaa914f461f099b535]) successfully deleted
I20260928 20:24:58.669323 920 ts_tablet_manager.cc:1905] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.669626 896 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Bootstrap starting.
I20260928 20:24:58.669986 896 tablet_bootstrap.cc:654] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.670056 897 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Bootstrap starting.
I20260928 20:24:58.670257 371 catalog_manager.cc:4841] TS 929de691888343a8878814cd4fe519cd (127.24.186.193:33033): tablet bcccf77a7e884b8a8581ac46f72c80b9 (table test-table [id=50c9739b24ed4ceaa914f461f099b535]) successfully deleted
I20260928 20:24:58.670315 522 tablet_service.cc:1467] Processing CreateTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 (DEFAULT_TABLE table=test-table [id=cce2b0b7115f44dd89bc32617a76ca7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:24:58.670419 522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b9f637965b4d8ba6b5d8bfafc884c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:24:58.670415 897 tablet_bootstrap.cc:654] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.670714 896 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: No bootstrap required, opened a new log
I20260928 20:24:58.670748 896 ts_tablet_manager.cc:1397] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.670835 896 raft_consensus.cc:348] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.670883 896 raft_consensus.cc:374] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.670897 896 raft_consensus.cc:729] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.670938 896 consensus_queue.cc:260] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.670977 920 ts_tablet_manager.cc:1918] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:24:58.671026 920 log.cc:1196] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/bcccf77a7e884b8a8581ac46f72c80b9
I20260928 20:24:58.671043 896 ts_tablet_manager.cc:1428] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 20:24:58.671240 920 ts_tablet_manager.cc:1939] T bcccf77a7e884b8a8581ac46f72c80b9 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting consensus metadata
I20260928 20:24:58.671985 371 catalog_manager.cc:4841] TS 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623): tablet bcccf77a7e884b8a8581ac46f72c80b9 (table test-table [id=50c9739b24ed4ceaa914f461f099b535]) successfully deleted
I20260928 20:24:58.672049 899 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: Bootstrap starting.
I20260928 20:24:58.672186 897 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: No bootstrap required, opened a new log
I20260928 20:24:58.672245 897 ts_tablet_manager.cc:1397] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:24:58.672349 897 raft_consensus.cc:348] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.672415 899 tablet_bootstrap.cc:654] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: Neither blocks nor log segments found. Creating new log.
I20260928 20:24:58.672453 897 raft_consensus.cc:374] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.672477 897 raft_consensus.cc:729] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9619eea451ef4070b4df8ba7ea11fc3d, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.672530 897 consensus_queue.cc:260] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.672647 897 ts_tablet_manager.cc:1428] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:58.673168 899 tablet_bootstrap.cc:492] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: No bootstrap required, opened a new log
I20260928 20:24:58.673210 899 ts_tablet_manager.cc:1397] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:24:58.673343 899 raft_consensus.cc:348] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.673414 899 raft_consensus.cc:374] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:24:58.673440 899 raft_consensus.cc:729] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929de691888343a8878814cd4fe519cd, State: Initialized, Role: FOLLOWER
I20260928 20:24:58.673484 899 consensus_queue.cc:260] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.673588 899 ts_tablet_manager.cc:1428] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:24:58.837010 954 raft_consensus.cc:482] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:24:58.837142 954 raft_consensus.cc:504] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.837276 954 leader_election.cc:288] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.837585 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b9f637965b4d8ba6b5d8bfafc884c4" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd" is_pre_election: true
I20260928 20:24:58.837585 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b9f637965b4d8ba6b5d8bfafc884c4" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" is_pre_election: true
I20260928 20:24:58.837697 542 raft_consensus.cc:2454] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.837697 822 raft_consensus.cc:2454] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 0.
I20260928 20:24:58.838030 615 leader_election.cc:302] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 9619eea451ef4070b4df8ba7ea11fc3d; no voters:
I20260928 20:24:58.838160 954 raft_consensus.cc:2781] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:24:58.838204 954 raft_consensus.cc:482] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:24:58.838225 954 raft_consensus.cc:3037] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.838924 954 raft_consensus.cc:504] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.839056 954 leader_election.cc:288] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Requested vote from peers 929de691888343a8878814cd4fe519cd (127.24.186.193:33033), 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623)
I20260928 20:24:58.839228 542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b9f637965b4d8ba6b5d8bfafc884c4" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "929de691888343a8878814cd4fe519cd"
I20260928 20:24:58.839301 542 raft_consensus.cc:3037] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.839247 822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b9f637965b4d8ba6b5d8bfafc884c4" candidate_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9619eea451ef4070b4df8ba7ea11fc3d"
I20260928 20:24:58.839342 822 raft_consensus.cc:3037] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:24:58.839978 542 raft_consensus.cc:2454] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.840131 822 raft_consensus.cc:2454] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1560a75b5b8c4d3bbf92f66f21fcca0f in term 1.
I20260928 20:24:58.840144 615 leader_election.cc:302] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1560a75b5b8c4d3bbf92f66f21fcca0f, 929de691888343a8878814cd4fe519cd; no voters:
I20260928 20:24:58.840242 954 raft_consensus.cc:2781] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:24:58.840291 954 raft_consensus.cc:686] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Becoming Leader. State: Replica: 1560a75b5b8c4d3bbf92f66f21fcca0f, State: Running, Role: LEADER
I20260928 20:24:58.840340 954 consensus_queue.cc:237] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } }
I20260928 20:24:58.841043 384 catalog_manager.cc:5495] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1560a75b5b8c4d3bbf92f66f21fcca0f (127.24.186.194). New cstate: current_term: 1 leader_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1560a75b5b8c4d3bbf92f66f21fcca0f" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 } health_report { overall_health: UNKNOWN } } }
I20260928 20:24:58.860139 542 raft_consensus.cc:1261] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.860108 822 raft_consensus.cc:1261] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Refusing update from remote peer 1560a75b5b8c4d3bbf92f66f21fcca0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:24:58.860420 954 consensus_queue.cc:1035] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9619eea451ef4070b4df8ba7ea11fc3d" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.860569 902 consensus_queue.cc:1035] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "929de691888343a8878814cd4fe519cd" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:24:58.870059 384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.870188 384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:24:58.870886 384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38fb2f80d62464782704486519a4de1: Sending DeleteTablet for 3 replicas of tablet d2b9f637965b4d8ba6b5d8bfafc884c4
I20260928 20:24:58.871124 522 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:41522
I20260928 20:24:58.871176 802 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37256
I20260928 20:24:58.871268 920 tablet_replica.cc:330] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: stopping tablet replica
I20260928 20:24:58.871260 919 tablet_replica.cc:330] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd: stopping tablet replica
I20260928 20:24:58.871307 920 raft_consensus.cc:2229] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.871341 920 raft_consensus.cc:2258] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.871328 919 raft_consensus.cc:2229] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:24:58.871376 919 raft_consensus.cc:2258] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 929de691888343a8878814cd4fe519cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.871429 662 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b9f637965b4d8ba6b5d8bfafc884c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:24:58 UTC) from {username='oryx', principal='oryx/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54840
I20260928 20:24:58.871522 918 tablet_replica.cc:330] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: stopping tablet replica
I20260928 20:24:58.871577 918 raft_consensus.cc:2229] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:24:58.871553 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 457
I20260928 20:24:58.871635 918 raft_consensus.cc:2258] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:24:58.872895 920 ts_tablet_manager.cc:1905] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.875300 920 ts_tablet_manager.cc:1918] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 20:24:58.875376 920 log.cc:1196] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/d2b9f637965b4d8ba6b5d8bfafc884c4
I20260928 20:24:58.875594 920 ts_tablet_manager.cc:1939] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 9619eea451ef4070b4df8ba7ea11fc3d: Deleting consensus metadata
I20260928 20:24:58.876175 371 catalog_manager.cc:4841] TS 9619eea451ef4070b4df8ba7ea11fc3d (127.24.186.195:45623): tablet d2b9f637965b4d8ba6b5d8bfafc884c4 (table test-table [id=cce2b0b7115f44dd89bc32617a76ca7f]) successfully deleted
W20260928 20:24:58.879494 371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:33033 (error 108)
I20260928 20:24:58.879524 918 ts_tablet_manager.cc:1905] T d2b9f637965b4d8ba6b5d8bfafc884c4 P 1560a75b5b8c4d3bbf92f66f21fcca0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:24:58.879698 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 597
I20260928 20:24:58.882218 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:24:58.881936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:33033 (local address 127.0.0.1:41538)
0928 20:24:58.882066 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.882165 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":47,"threads_started":1}
W20260928 20:24:58.882349 371 catalog_manager.cc:4584] TS 929de691888343a8878814cd4fe519cd (127.24.186.193:33033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2b9f637965b4d8ba6b5d8bfafc884c4: Network error: Client connection negotiation failed: client connection to 127.24.186.193:33033: connect: Connection refused (error 111)
W20260928 20:24:58.886600 987 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:24:58.886424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46241 (local address 127.0.0.1:54858)
0928 20:24:58.886513 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 20:24:58.886562 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.194:46241: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 20:24:58.887099 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 737
I20260928 20:24:58.893421 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 353
2026-09-28T20:24:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3057 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[990](info): set up 2 sockets
Sep 28 20:24:58 dist-test-slave-k477 krb5kdc[990](info): commencing operation
krb5kdc: starting...
W20260928 20:24:59.969091 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:25:00 dist-test-slave-k477 krb5kdc[990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:25:00.095474 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:45031
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:00.184067 1003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:00.184309 1003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:00.184334 1003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:00.185724 1003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:00.185786 1003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:00.185801 1003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:00.185813 1003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:00.185827 1003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:00.187695 1003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:45031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:00.188112 1003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:00.188689 1003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:00.194798 1009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:00.194960 1011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:00.195098 1008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:00.195214 1003 server_base.cc:1034] running on GCE node
I20260928 20:25:00.195640 1003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:00.196071 1003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:00.197306 1003 hybrid_clock.cc:648] HybridClock initialized: now 1790627100197262 us; error 59 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T20:25:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1306 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:25:00 dist-test-slave-k477 krb5kdc[1016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:25:00 dist-test-slave-k477 krb5kdc[1016](info): set up 2 sockets
Sep 28 20:25:00 dist-test-slave-k477 krb5kdc[1016](info): commencing operation
krb5kdc: starting...
W20260928 20:25:01.281071 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:25:01.415405 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41579
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:01.505870 1029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:01.506155 1029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:01.506198 1029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:01.507442 1029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:01.507520 1029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:01.507550 1029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:01.507582 1029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:01.507611 1029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:01.509189 1029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:01.509490 1029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:01.509737 1029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:01.515566 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.515569 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.515570 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:01.515607 1029 server_base.cc:1034] running on GCE node
I20260928 20:25:01.516279 1029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:01.516568 1029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:01.517747 1029 hybrid_clock.cc:648] HybridClock initialized: now 1790627101517698 us; error 64 us; skew 500 ppm
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:01.520433 1029 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:25:01.520903 1029 webserver.cc:460] Webserver started at http://127.24.186.254:46111/ using document root <none> and password file <none>
I20260928 20:25:01.521153 1029 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:01.521224 1029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:01.521385 1029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:01.522500 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "98b86e51260645a6a1c7a744b8534b32"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "d373766ce9adbf8336a36e8ca8906a77"
server_key_iv: "e4f7c3dbda28d7e49f12a71f228845c7"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.522889 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "98b86e51260645a6a1c7a744b8534b32"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "d373766ce9adbf8336a36e8ca8906a77"
server_key_iv: "e4f7c3dbda28d7e49f12a71f228845c7"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.525292 1029 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:25:01.526216 1044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.526432 1029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:01.526553 1029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "98b86e51260645a6a1c7a744b8534b32"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "d373766ce9adbf8336a36e8ca8906a77"
server_key_iv: "e4f7c3dbda28d7e49f12a71f228845c7"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.526643 1029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:01.550479 1029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:01.552588 1029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:01.552783 1029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:01.557032 1029 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41579
I20260928 20:25:01.557066 1096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41579 every 8 connection(s)
I20260928 20:25:01.557451 1029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:25:01.558314 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:01.560770 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Bootstrap starting.
I20260928 20:25:01.561053 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1029
I20260928 20:25:01.561183 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:25:01.561363 25323 external_mini_cluster.cc:1428] Setting key f9595c46c38795a91c8944a682ba405d
I20260928 20:25:01.561539 1097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:01.561851 1097 log.cc:824] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:01.562795 1097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: No bootstrap required, opened a new log
I20260928 20:25:01.564715 1097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:01.564889 1097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:01.564935 1097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b86e51260645a6a1c7a744b8534b32, State: Initialized, Role: FOLLOWER
I20260928 20:25:01.565085 1097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:01.565181 1097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:25:01.565214 1097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:25:01.565265 1097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:01.566020 1097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:01.566125 1097 leader_election.cc:302] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98b86e51260645a6a1c7a744b8534b32; no voters:
I20260928 20:25:01.566334 1097 leader_election.cc:288] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:25:01.566392 1102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:25:01.566557 1102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 LEADER]: Becoming Leader. State: Replica: 98b86e51260645a6a1c7a744b8534b32, State: Running, Role: LEADER
I20260928 20:25:01.566632 1097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:25:01.566658 1102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:01.566985 1102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b86e51260645a6a1c7a744b8534b32. Latest consensus state: current_term: 1 leader_uuid: "98b86e51260645a6a1c7a744b8534b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } } }
I20260928 20:25:01.566993 1103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98b86e51260645a6a1c7a744b8534b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } } }
I20260928 20:25:01.567076 1102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:01.567101 1103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:01.567478 1107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:25:01.568081 1107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:01.570145 1107 catalog_manager.cc:1347] Generated new cluster ID: 9e41362373a1431cac8a5d29ae5d2a43
I20260928 20:25:01.570354 1107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:25:01.573877 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:01.562073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.24.186.254:41579)
0928 20:25:01.562285 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:01.562292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:01.562320 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:25:01.562750 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:01.562756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:01.562801 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:01.563012 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:01.563858 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.564901 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.566027 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.566750 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.567200 (+ 450us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:01.569924 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:01.569928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:01.569929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:01.569943 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:01.571973 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.572363 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.572366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.572366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.572466 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.572634 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.572634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.572635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.572919 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:01.573059 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:01.573536 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 20:25:01.573670 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260928 20:25:01.583288 1107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:25:01.584003 1107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:25:01.591755 1107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Generated new TSK 0
I20260928 20:25:01.592060 1107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:25:01.630743 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:01.719658 1123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:01.719936 1123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:01.719964 1123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:01.721657 1123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:01.721738 1123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:01.721769 1123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:25:01.723774 1123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:01.724143 1123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:01.724401 1123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:01.730363 1131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.730408 1129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.730362 1128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:01.730502 1123 server_base.cc:1034] running on GCE node
I20260928 20:25:01.730818 1123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:01.731102 1123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:01.732268 1123 hybrid_clock.cc:648] HybridClock initialized: now 1790627101732254 us; error 46 us; skew 500 ppm
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:01.735294 1123 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:25:01.735922 1123 webserver.cc:460] Webserver started at http://127.24.186.193:45759/ using document root <none> and password file <none>
I20260928 20:25:01.736708 1123 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:01.736872 1123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:01.737068 1123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:01.738441 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "574379810a084d279d299286275949c1"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "b7556567821c5f4bb0c65505105a5f6d"
server_key_iv: "e7a818e9bb1d0a63a8e571dc81e4f074"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.738876 1123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "574379810a084d279d299286275949c1"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "b7556567821c5f4bb0c65505105a5f6d"
server_key_iv: "e7a818e9bb1d0a63a8e571dc81e4f074"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.740394 1123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 20:25:01.741277 1138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.741588 1123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:01.741705 1123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "574379810a084d279d299286275949c1"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "b7556567821c5f4bb0c65505105a5f6d"
server_key_iv: "e7a818e9bb1d0a63a8e571dc81e4f074"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.741796 1123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:01.765126 1123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:01.767970 1123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:01.768101 1123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:01.768433 1123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:01.768801 1123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:01.768862 1123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.768906 1123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:01.768934 1123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.775353 1123 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:43853
I20260928 20:25:01.775431 1251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:43853 every 8 connection(s)
I20260928 20:25:01.775774 1123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:25:01.776917 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1123
I20260928 20:25:01.777064 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:25:01.777307 25323 external_mini_cluster.cc:1428] Setting key 9d7f4f4da83675619aec7f2f3a707547
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:25:01.787755 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:01.776742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:58881 (local address 127.24.186.254:41579)
0928 20:25:01.776866 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:01.776870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:01.776989 (+ 119us) server_negotiation.cc:407] Connection header received
0928 20:25:01.777734 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:01.777736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:01.777763 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:01.777853 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:01.779213 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.780010 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.781075 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.781318 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.781404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:01.784030 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:01.784032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:01.784033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:01.784043 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:01.785826 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.786289 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.786290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.786290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.786356 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.786756 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.786757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.786758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.786913 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:01.787180 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:01.787537 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 20:25:01.787672 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 20:25:01.787955 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:01.776552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:58881)
0928 20:25:01.776841 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:01.776889 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:01.777529 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:01.778131 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:01.778136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:01.778874 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:01.779079 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:01.779084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.780130 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.780131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:01.780958 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:01.780960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.781413 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.781414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:01.781549 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:01.782149 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:01.782158 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:01.783928 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:01.785925 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:01.785937 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:01.785940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:01.786202 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:01.786429 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:01.786430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:01.786430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:01.786605 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:01.787213 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:01.787220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:01.787375 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 20:25:01.787517 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 20:25:01.787701 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
I20260928 20:25:01.788575 1252 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41579
I20260928 20:25:01.788684 1252 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:01.788885 1252 heartbeater.cc:511] Master 127.24.186.254:41579 requested a full tablet report, sending...
I20260928 20:25:01.789582 1061 ts_manager.cc:194] Registered new tserver with Master: 574379810a084d279d299286275949c1 (127.24.186.193:43853)
I20260928 20:25:01.790462 1061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:58881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:25:01.806068 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:01.891752 1257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:01.892014 1257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:01.892041 1257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:01.893349 1257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:01.893391 1257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:01.893420 1257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:25:01.894879 1257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:01.895079 1257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:01.895278 1257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:01.900331 1262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.900333 1265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:01.900354 1263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:01.900919 1257 server_base.cc:1034] running on GCE node
I20260928 20:25:01.901158 1257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:01.901366 1257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:01.902562 1257 hybrid_clock.cc:648] HybridClock initialized: now 1790627101902542 us; error 28 us; skew 500 ppm
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:01.905216 1257 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:25:01.905665 1257 webserver.cc:460] Webserver started at http://127.24.186.194:35397/ using document root <none> and password file <none>
I20260928 20:25:01.905905 1257 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:01.905973 1257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:01.906134 1257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:01.907254 1257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "9c0981c310f14131a2a12291902e8dd7"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "f424a8d41aac2747f4235df59cdb76de"
server_key_iv: "9712cdb84959af287616b2e4581ba172"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.907634 1257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "9c0981c310f14131a2a12291902e8dd7"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "f424a8d41aac2747f4235df59cdb76de"
server_key_iv: "9712cdb84959af287616b2e4581ba172"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.909227 1257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:01.910308 1272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.910532 1257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:01.910641 1257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "9c0981c310f14131a2a12291902e8dd7"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "f424a8d41aac2747f4235df59cdb76de"
server_key_iv: "9712cdb84959af287616b2e4581ba172"
server_key_version: "encryptionkey@0"
I20260928 20:25:01.910730 1257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:01.926359 1257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:01.928623 1257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:01.928830 1257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:01.929150 1257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:01.929504 1257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:01.929564 1257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.929606 1257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:01.929634 1257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:01.935724 1257 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:41445
I20260928 20:25:01.935849 1385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:41445 every 8 connection(s)
I20260928 20:25:01.936188 1257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:25:01.941603 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1257
I20260928 20:25:01.941748 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:25:01.941979 25323 external_mini_cluster.cc:1428] Setting key de0e82fe30860d6dde0977dfb6f15cf4
Sep 28 20:25:01 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627101, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:01.947168 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:01.936980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:39939 (local address 127.24.186.254:41579)
0928 20:25:01.937060 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:01.937062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:01.937177 (+ 115us) server_negotiation.cc:407] Connection header received
0928 20:25:01.937711 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:01.937712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:01.937724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:01.937800 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:01.939499 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.940066 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.941234 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.941428 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.941492 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:01.944276 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:01.944278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:01.944278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:01.944291 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:01.945732 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.946167 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.946168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.946169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.946228 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:01.946485 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:01.946486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:01.946486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:01.946619 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:01.946694 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:01.946984 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 20:25:01.947069 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 20:25:01.947379 1388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:01.936886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:39939)
0928 20:25:01.937043 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:01.937091 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:01.937585 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:01.938066 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:01.938073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:01.939044 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:01.939325 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:01.939338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.940225 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.940227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:01.941125 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:01.941128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:01.941542 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:01.941544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:01.941711 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:01.942434 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:01.942443 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:01.944144 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:01.945836 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:01.945844 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:01.945847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:01.946087 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:01.946300 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:01.946301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:01.946302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:01.946416 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:01.946684 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:01.946691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:01.946848 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 20:25:01.946962 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 20:25:01.947134 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 20:25:01.947947 1386 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41579
I20260928 20:25:01.948064 1386 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:01.948304 1386 heartbeater.cc:511] Master 127.24.186.254:41579 requested a full tablet report, sending...
I20260928 20:25:01.948719 1061 ts_manager.cc:194] Registered new tserver with Master: 9c0981c310f14131a2a12291902e8dd7 (127.24.186.194:41445)
I20260928 20:25:01.949200 1061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:39939
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:25:01.971302 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:02.061655 1391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:02.061903 1391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:02.061930 1391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:02.063210 1391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:02.063257 1391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:02.063314 1391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:25:02.064878 1391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:02.065145 1391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:02.065379 1391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:02.071717 1397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.071943 1391 server_base.cc:1034] running on GCE node
W20260928 20:25:02.071717 1396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.071719 1399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.072458 1391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:02.072702 1391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:02.073905 1391 hybrid_clock.cc:648] HybridClock initialized: now 1790627102073872 us; error 51 us; skew 500 ppm
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:02.077210 1391 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:25:02.078116 1391 webserver.cc:460] Webserver started at http://127.24.186.195:35387/ using document root <none> and password file <none>
I20260928 20:25:02.078408 1391 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:02.078464 1391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:02.078605 1391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:02.079721 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "2d1b0884b1044e968687d9fd8fe5e1f0"
format_stamp: "Formatted at 2026-09-28 20:25:02 on dist-test-slave-k477"
server_key: "355f53af1aa9887fb59bb26875b2c822"
server_key_iv: "de4bdaf145100d178fdbdaa368c18af5"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.080185 1391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "2d1b0884b1044e968687d9fd8fe5e1f0"
format_stamp: "Formatted at 2026-09-28 20:25:02 on dist-test-slave-k477"
server_key: "355f53af1aa9887fb59bb26875b2c822"
server_key_iv: "de4bdaf145100d178fdbdaa368c18af5"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.082262 1391 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 20:25:02.083137 1406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.083365 1391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.083462 1391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "2d1b0884b1044e968687d9fd8fe5e1f0"
format_stamp: "Formatted at 2026-09-28 20:25:02 on dist-test-slave-k477"
server_key: "355f53af1aa9887fb59bb26875b2c822"
server_key_iv: "de4bdaf145100d178fdbdaa368c18af5"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.083541 1391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:02.096652 1391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:02.099311 1391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:02.099606 1391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:02.100051 1391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:02.100605 1391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:02.100718 1391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.100783 1391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:02.100816 1391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.108543 1391 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:42195
I20260928 20:25:02.108592 1519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:42195 every 8 connection(s)
I20260928 20:25:02.108970 1391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:02.117723 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1391
I20260928 20:25:02.117923 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:25:02.118153 25323 external_mini_cluster.cc:1428] Setting key 1f7579853083a2559fb198425f98e208
I20260928 20:25:02.121011 1100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.110133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:46071 (local address 127.24.186.254:41579)
0928 20:25:02.110188 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.110190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.110221 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:25:02.110868 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.110869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.110880 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.110939 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.112151 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.112648 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.113906 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.114148 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.114212 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.117341 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:02.117343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:02.117343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:02.117351 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:02.119020 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:02.119513 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:02.119514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:02.119514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:02.119569 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:02.120093 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:02.120094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:02.120094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:02.120230 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:02.120319 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:02.120739 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 20:25:02.120919 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:25:02.121171 1522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.109899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:46071)
0928 20:25:02.110113 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.110152 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.110640 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.111046 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.111050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.111782 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.112051 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.112057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.112737 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.112738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.113728 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.113731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.114280 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.114282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.114473 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.115387 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.115397 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.117233 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:02.119107 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:02.119116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:02.119119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:02.119450 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:02.119712 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:02.119713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:02.119713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:02.119992 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:02.120331 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:02.120337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:02.120479 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 20:25:02.120574 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 20:25:02.120719 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":79,"threads_started":1}
I20260928 20:25:02.121878 1520 heartbeater.cc:347] Connected to a master server at 127.24.186.254:41579
I20260928 20:25:02.121994 1520 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:02.122197 1520 heartbeater.cc:511] Master 127.24.186.254:41579 requested a full tablet report, sending...
I20260928 20:25:02.122649 1061 ts_manager.cc:194] Registered new tserver with Master: 2d1b0884b1044e968687d9fd8fe5e1f0 (127.24.186.195:42195)
I20260928 20:25:02.123065 1061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:46071
I20260928 20:25:02.130345 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
Entry for principal oryx/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab.
WARNING: no policy specified for oryx/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
Entry for principal oryx/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab.
WARNING: no policy specified for oryx/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
Entry for principal oryx/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab.
I20260928 20:25:02.208611 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1123
I20260928 20:25:02.213778 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1257
I20260928 20:25:02.218189 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1391
I20260928 20:25:02.222818 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1029
I20260928 20:25:02.229780 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41579
--webserver_interface=127.24.186.254
--webserver_port=46111
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:02.319594 1529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:02.319903 1529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:02.319959 1529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:02.321215 1529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:02.321290 1529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:02.321314 1529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:02.321343 1529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:02.321374 1529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:02.322959 1529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=46111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:02.323283 1529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:02.323539 1529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:02.328575 1535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.328549 1537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.328764 1529 server_base.cc:1034] running on GCE node
W20260928 20:25:02.328887 1534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.329180 1529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:02.329550 1529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:02.330708 1529 hybrid_clock.cc:648] HybridClock initialized: now 1790627102330693 us; error 27 us; skew 500 ppm
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:02.333446 1529 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:25:02.334044 1529 webserver.cc:460] Webserver started at http://127.24.186.254:46111/ using document root <none> and password file <none>
I20260928 20:25:02.334303 1529 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:02.334367 1529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:02.335871 1529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:25:02.336552 1544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.336763 1529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.336858 1529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "98b86e51260645a6a1c7a744b8534b32"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "d373766ce9adbf8336a36e8ca8906a77"
server_key_iv: "e4f7c3dbda28d7e49f12a71f228845c7"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.337222 1529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:02.358676 1529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:02.360997 1529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:02.361166 1529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:02.365767 1529 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41579
I20260928 20:25:02.365803 1596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41579 every 8 connection(s)
I20260928 20:25:02.366204 1529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:25:02.366384 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1529
I20260928 20:25:02.366722 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:43853
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:41579
--webserver_port=45759
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:25:02.367872 1597 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:25:02.368572 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Bootstrap starting.
I20260928 20:25:02.378350 1597 log.cc:824] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:02.380337 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:25:02.380631 1597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Bootstrap complete.
I20260928 20:25:02.382939 1597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:02.383203 1597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98b86e51260645a6a1c7a744b8534b32, State: Initialized, Role: FOLLOWER
I20260928 20:25:02.383409 1597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:02.383505 1597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:25:02.383558 1597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:25:02.383607 1597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:25:02.384693 1597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:02.384814 1597 leader_election.cc:302] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98b86e51260645a6a1c7a744b8534b32; no voters:
I20260928 20:25:02.385005 1597 leader_election.cc:288] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:25:02.385082 1601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:25:02.385241 1601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [term 2 LEADER]: Becoming Leader. State: Replica: 98b86e51260645a6a1c7a744b8534b32, State: Running, Role: LEADER
I20260928 20:25:02.385282 1597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:25:02.385346 1601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } }
I20260928 20:25:02.385721 1602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98b86e51260645a6a1c7a744b8534b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } } }
I20260928 20:25:02.385815 1602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:02.386068 1603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98b86e51260645a6a1c7a744b8534b32. Latest consensus state: current_term: 2 leader_uuid: "98b86e51260645a6a1c7a744b8534b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98b86e51260645a6a1c7a744b8534b32" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41579 } } }
I20260928 20:25:02.386122 1611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:25:02.386775 1611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:25:02.386930 1611 catalog_manager.cc:1259] Loaded cluster ID: 9e41362373a1431cac8a5d29ae5d2a43
I20260928 20:25:02.387125 1603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:02.387209 1611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:25:02.388303 1611 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:25:02.388485 1611 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98b86e51260645a6a1c7a744b8534b32: Loaded TSK: 0
I20260928 20:25:02.388697 1611 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:25:02.462277 1599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:02.462491 1599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:02.462519 1599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:02.463757 1599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:02.463901 1599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:02.463961 1599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:25:02.466084 1599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.193.keytab
--principal=oryx/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:43853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=45759
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:02.466615 1599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:02.466955 1599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:02.473217 1626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.473240 1624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.473347 1623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.473459 1599 server_base.cc:1034] running on GCE node
I20260928 20:25:02.473821 1599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:02.474097 1599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:02.475291 1599 hybrid_clock.cc:648] HybridClock initialized: now 1790627102475274 us; error 31 us; skew 500 ppm
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:02.478330 1599 init.cc:377] Logged in from keytab as oryx/127.24.186.193@KRBTEST.COM (short username oryx)
I20260928 20:25:02.478838 1599 webserver.cc:460] Webserver started at http://127.24.186.193:45759/ using document root <none> and password file <none>
I20260928 20:25:02.479084 1599 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:02.479175 1599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:02.480695 1599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:02.481410 1633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.481645 1599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:02.481762 1599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "574379810a084d279d299286275949c1"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "b7556567821c5f4bb0c65505105a5f6d"
server_key_iv: "e7a818e9bb1d0a63a8e571dc81e4f074"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.482201 1599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:02.500375 1599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:02.502767 1599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:02.502943 1599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:02.503209 1599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:02.503523 1599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:02.503578 1599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.503628 1599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:02.503676 1599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.510057 1599 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:43853
I20260928 20:25:02.510102 1746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:43853 every 8 connection(s)
I20260928 20:25:02.510442 1599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:25:02.512557 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1599
I20260928 20:25:02.512944 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:41445
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:41579
--webserver_port=35397
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.521020 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.511225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:36681)
0928 20:25:02.511401 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.511443 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.511988 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.512494 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.512498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.513391 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.513703 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.513710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.515422 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.515423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.516332 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.516335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.517132 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.517133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.517290 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.517962 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.517971 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.520793 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
W20260928 20:25:02.521463 1747 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:41579 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.186.254:41579: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
W20260928 20:25:02.522121 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.511306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36681 (local address 127.24.186.254:41579)
0928 20:25:02.511517 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.511525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.511571 (+ 46us) server_negotiation.cc:407] Connection header received
0928 20:25:02.512196 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.512203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.512258 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.512409 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.513865 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.515276 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.516435 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.517034 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.517160 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.520991 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:36681: BlockingRecv error: recv got EOF from 127.24.186.193:36681 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:25:02.528245 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.521814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:34613 (local address 127.24.186.254:41579)
0928 20:25:02.522282 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.522284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.522294 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:25:02.522322 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.522323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.522336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.522403 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.522747 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.523575 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.524636 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.524974 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.525041 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.528168 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:34613: BlockingRecv error: recv got EOF from 127.24.186.193:34613 (error 108)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":90,"threads_started":1}
I20260928 20:25:02.528558 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.521748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:34613)
0928 20:25:02.521804 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.521815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.521856 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.522425 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.522426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.522580 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.522641 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.522643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.523687 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.523688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.524542 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.524545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.525102 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.525103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.525177 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.525725 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.525731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.528479 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.534735 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.528936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:42067)
0928 20:25:02.529071 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.529082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.529131 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.529282 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.529283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.529435 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.529480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.529482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.530594 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.530595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.531467 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.531470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.531988 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.531989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.532048 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.532533 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.532538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.534653 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260928 20:25:02.534873 1747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:25:02.535032 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.529006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:42067 (local address 127.24.186.254:41579)
0928 20:25:02.529048 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.529050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.529134 (+ 84us) server_negotiation.cc:407] Connection header received
0928 20:25:02.529185 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.529185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.529197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.529263 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.529554 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.530493 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.531570 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.531899 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.531967 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.534767 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:42067: BlockingRecv error: recv got EOF from 127.24.186.193:42067 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:25:02.604130 1751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:02.604396 1751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:02.604439 1751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:02.606367 1751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:02.606542 1751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:02.606611 1751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:25:02.609496 1751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.194.keytab
--principal=oryx/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:41445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=35397
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:02.609958 1751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:02.610241 1751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:02.616058 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.616153 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.616262 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.616516 1751 server_base.cc:1034] running on GCE node
I20260928 20:25:02.616858 1751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:02.617215 1751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:02.618371 1751 hybrid_clock.cc:648] HybridClock initialized: now 1790627102618345 us; error 37 us; skew 500 ppm
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:02.621055 1751 init.cc:377] Logged in from keytab as oryx/127.24.186.194@KRBTEST.COM (short username oryx)
I20260928 20:25:02.621502 1751 webserver.cc:460] Webserver started at http://127.24.186.194:35397/ using document root <none> and password file <none>
I20260928 20:25:02.621747 1751 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:02.621832 1751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:02.623273 1751 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.624107 1768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.624374 1751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.624495 1751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "9c0981c310f14131a2a12291902e8dd7"
format_stamp: "Formatted at 2026-09-28 20:25:01 on dist-test-slave-k477"
server_key: "f424a8d41aac2747f4235df59cdb76de"
server_key_iv: "9712cdb84959af287616b2e4581ba172"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.624886 1751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:02.645953 1751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:02.648180 1751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:02.648351 1751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:02.648586 1751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:02.648919 1751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:02.648976 1751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.649019 1751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:02.649046 1751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.656176 1751 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:41445
I20260928 20:25:02.656219 1881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:41445 every 8 connection(s)
I20260928 20:25:02.656589 1751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:25:02.659682 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1751
I20260928 20:25:02.660018 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:42195
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:41579
--webserver_port=35387
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:25:02.664861 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.657483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:49333 (local address 127.24.186.254:41579)
0928 20:25:02.657548 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.657551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.657825 (+ 274us) server_negotiation.cc:407] Connection header received
0928 20:25:02.658264 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.658265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.658279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.658336 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.659292 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.659896 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.660992 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.661322 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.661400 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.664802 (+ 3402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:49333: BlockingRecv error: recv got EOF from 127.24.186.194:49333 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:25:02.664939 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.657482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:49333)
0928 20:25:02.657655 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.657692 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.658177 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.658404 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.658407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.659002 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.659199 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.659203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.659997 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.659998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.660888 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.660890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.661426 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.661427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.661559 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.662158 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.662166 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.664716 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
W20260928 20:25:02.665169 1882 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:41579 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.186.254:41579: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.669775 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.665426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:41149)
0928 20:25:02.665477 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.665486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.665543 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.665690 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.665690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.665823 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.665873 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.665874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.666498 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.666499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.666992 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.666993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.667375 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.667376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.667447 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.667910 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.667920 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.669688 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 20:25:02.669826 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.665471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:41149 (local address 127.24.186.254:41579)
0928 20:25:02.665534 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.665536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.665545 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:02.665609 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.665610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.665619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.665681 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.665955 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.666421 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.667084 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.667289 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.667374 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.669762 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:41149: BlockingRecv error: recv got EOF from 127.24.186.194:41149 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.677243 1884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.670039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:40801)
0928 20:25:02.670078 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.670094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.670156 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.670306 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.670307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.670428 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.670456 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.670457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.671071 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.671072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.671562 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.671564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.672155 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.672156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.672215 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.673049 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.673058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.677140 (+ 4082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 20:25:02.677546 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.670089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40801 (local address 127.24.186.254:41579)
0928 20:25:02.670145 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.670148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.670156 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:02.670236 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.670236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.670249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.670294 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.670534 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.670993 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.671659 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.672059 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.672140 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.677271 (+ 5131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:40801: BlockingRecv error: recv got EOF from 127.24.186.194:40801 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 20:25:02.679473 1882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:25:02.751890 1885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:02.752123 1885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:02.752151 1885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:02.753494 1885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:02.753567 1885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:02.753604 1885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:25:02.755192 1885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/krb5kdc/oryx_127.24.186.195.keytab
--principal=oryx/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:42195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=35387
--tserver_master_addrs=127.24.186.254:41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:02.755443 1885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:02.755645 1885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:25:02.760939 1885 server_base.cc:1034] running on GCE node
W20260928 20:25:02.760854 1890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.760859 1893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:02.760859 1891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:02.761307 1885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:02.761631 1885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:02.762817 1885 hybrid_clock.cc:648] HybridClock initialized: now 1790627102762797 us; error 35 us; skew 500 ppm
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627102, etypes {rep=17 tkt=17 ses=17}, oryx/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:02.772558 1885 init.cc:377] Logged in from keytab as oryx/127.24.186.195@KRBTEST.COM (short username oryx)
I20260928 20:25:02.773048 1885 webserver.cc:460] Webserver started at http://127.24.186.195:35387/ using document root <none> and password file <none>
I20260928 20:25:02.773298 1885 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:02.773382 1885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:02.775141 1885 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.776227 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.776472 1885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:02.776602 1885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "2d1b0884b1044e968687d9fd8fe5e1f0"
format_stamp: "Formatted at 2026-09-28 20:25:02 on dist-test-slave-k477"
server_key: "355f53af1aa9887fb59bb26875b2c822"
server_key_iv: "de4bdaf145100d178fdbdaa368c18af5"
server_key_version: "encryptionkey@0"
I20260928 20:25:02.777009 1885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:02.797294 1885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:02.799665 1885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:02.799909 1885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:02.800171 1885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:02.800484 1885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:02.800539 1885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.800585 1885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:02.800608 1885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:02.806846 1885 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:42195
I20260928 20:25:02.806886 2013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:42195 every 8 connection(s)
I20260928 20:25:02.807256 1885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.816809 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 1885
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 20:25:02.817893 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.808048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:45297 (local address 127.24.186.254:41579)
0928 20:25:02.808102 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.808105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.808304 (+ 199us) server_negotiation.cc:407] Connection header received
0928 20:25:02.808858 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.808860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.808872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.808926 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.809903 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.810399 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.811393 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.811603 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.811691 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.817820 (+ 6129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:45297: BlockingRecv error: recv got EOF from 127.24.186.195:45297 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 20:25:02.818200 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.808001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:45297)
0928 20:25:02.808170 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.808213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.808724 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.809004 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.809008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.809588 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.809781 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.809786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.810509 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.810510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.811297 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.811299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.811908 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.811909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.812012 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.814659 (+ 2647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.814671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.817740 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
W20260928 20:25:02.818487 2014 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:41579 (0 consecutive failures): Not authorized: Failed to ping master at 127.24.186.254:41579: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.823666 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.818741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:50115)
0928 20:25:02.818810 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.818821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.818870 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.818994 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.818995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.819131 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.819177 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.819178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.819885 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.819887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.820413 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.820415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.820828 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.820829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.820896 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.821561 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.821573 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.823556 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 20:25:02.823693 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.818772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:50115 (local address 127.24.186.254:41579)
0928 20:25:02.818813 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.818815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.818855 (+ 40us) server_negotiation.cc:407] Connection header received
0928 20:25:02.818920 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.818921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.818930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.818993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.819257 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.819762 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.820516 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.820734 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.820833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.823647 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:50115: BlockingRecv error: recv got EOF from 127.24.186.195:50115 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 20:25:02.826813 1752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.817859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.24.186.254:41579)
0928 20:25:02.817918 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.817919 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.817925 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:25:02.817949 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.817950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.817978 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.818040 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.818894 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.819861 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.821125 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.821428 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.821483 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.823061 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:02.823067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:02.823068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:02.823083 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:02.825252 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:02.825693 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:02.825697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:02.825698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:02.825821 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:02.826076 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:02.826077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:02.826077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:02.826323 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:02.826472 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:02.826582 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:25:02.826696 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:02 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:02.831899 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:02.824549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:48649)
0928 20:25:02.824646 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:02.824657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:02.824697 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:02.825439 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:02.825440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:02.825602 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:02.825659 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.825662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.826671 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.826672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.827647 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:02.827651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.828269 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.828270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:02.828343 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.828746 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:02.828751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:02.831651 (+ 2900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 20:25:02.831955 1750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:02.824657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:48649 (local address 127.24.186.254:41579)
0928 20:25:02.825334 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:02.825336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:02.825340 (+ 4us) server_negotiation.cc:407] Connection header received
0928 20:25:02.825356 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:02.825356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:02.825363 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:02.825430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:02.825728 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.826572 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.827755 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:02.828147 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:02.828192 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:02.831911 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:48649: BlockingRecv error: recv got EOF from 127.24.186.195:48649 (error 108)
Metrics: {"server-negotiator.queue_time_us":661}
W20260928 20:25:02.832240 2014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:03.540664 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:03.535490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:52447)
0928 20:25:03.535685 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:03.535697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:03.535781 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:03.536156 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:03.536158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:03.536348 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:03.536404 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.536405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.537126 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.537127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.537679 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.537681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.538012 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.538013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.538056 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.538494 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:03.538498 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:03.540594 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
W20260928 20:25:03.540876 2019 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:03.535543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52447 (local address 127.24.186.254:41579)
0928 20:25:03.535688 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:03.535692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:03.535744 (+ 52us) server_negotiation.cc:407] Connection header received
0928 20:25:03.535996 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:03.535999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:03.536025 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:03.536129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:03.536519 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.537043 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.537759 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.537943 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.537992 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.540791 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:52447: BlockingRecv error: recv got EOF from 127.24.186.193:52447 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:03.685526 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:03.679981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:55813)
0928 20:25:03.680101 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:03.680116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:03.680186 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:03.680356 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:03.680357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:03.680551 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:03.680598 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.680600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.681283 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.681284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.681967 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.681971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.682477 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.682478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.682527 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.683067 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:03.683075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:03.685413 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260928 20:25:03.685627 2019 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:03.680067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:55813 (local address 127.24.186.254:41579)
0928 20:25:03.680120 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:03.680123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:03.680168 (+ 45us) server_negotiation.cc:407] Connection header received
0928 20:25:03.680262 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:03.680263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:03.680276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:03.680331 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:03.680690 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.681205 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.682077 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.682354 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.682432 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.685552 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:55813: BlockingRecv error: recv got EOF from 127.24.186.194:55813 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:03 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:03.838555 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:03.832747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:53275)
0928 20:25:03.832907 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:03.832919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:03.832974 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:03.833445 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:03.833447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:03.833696 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:03.833766 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.833768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.834724 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.834727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.835577 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:03.835580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.836267 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.836268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:03.836317 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.836753 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:03.836766 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:03.838490 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260928 20:25:03.838708 2019 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:03.833074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:53275 (local address 127.24.186.254:41579)
0928 20:25:03.833201 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:03.833203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:03.833218 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:25:03.833258 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:03.833260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:03.833277 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:03.833423 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:03.833872 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.834451 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.835671 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:03.836183 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:03.836243 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:03.838609 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:53275: BlockingRecv error: recv got EOF from 127.24.186.195:53275 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:04.545797 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:04.541197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:35593)
0928 20:25:04.541340 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:04.541352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:04.541413 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:04.541566 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:04.541567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:04.541744 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:04.541794 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.541795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.542495 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.542496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.543033 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.543035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.543394 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.543395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.543436 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.544020 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:04.544025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:04.545733 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260928 20:25:04.545917 2023 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:04.541282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:35593 (local address 127.24.186.254:41579)
0928 20:25:04.541418 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:04.541420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:04.541455 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:25:04.541480 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:04.541481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:04.541491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:04.541542 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:04.541870 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.542400 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.543119 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.543326 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.543381 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.545832 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:35593: BlockingRecv error: recv got EOF from 127.24.186.193:35593 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:04.691313 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:04.686254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:49699)
0928 20:25:04.686511 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:04.686525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:04.686641 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:04.686827 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:04.686829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:04.687083 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:04.687146 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.687148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.687862 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.687864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.688411 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.688414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.688793 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.688794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.688850 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.689374 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:04.689380 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:04.691238 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":205,"threads_started":1}
W20260928 20:25:04.691443 2023 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:04.686438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:49699 (local address 127.24.186.254:41579)
0928 20:25:04.686502 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:04.686505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:04.686617 (+ 112us) server_negotiation.cc:407] Connection header received
0928 20:25:04.686723 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:04.686724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:04.686737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:04.686820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:04.687227 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.687740 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.688505 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.688711 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.688794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.691340 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:49699: BlockingRecv error: recv got EOF from 127.24.186.194:49699 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:04 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:04.843761 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:04.839164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:46663)
0928 20:25:04.839417 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:04.839432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:04.839494 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:04.839652 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:04.839653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:04.839857 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:04.839906 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.839908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.840583 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.840584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.841175 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:04.841176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.841519 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.841519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:04.841557 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.842000 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:04.842004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:04.843702 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":70,"threads_started":1}
W20260928 20:25:04.843955 2023 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:04.839389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:46663 (local address 127.24.186.254:41579)
0928 20:25:04.839463 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:04.839465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:04.839480 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:25:04.839551 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:04.839551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:04.839563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:04.839626 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:04.839983 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.840510 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.841260 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:04.841442 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:04.841508 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:04.843877 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:46663: BlockingRecv error: recv got EOF from 127.24.186.195:46663 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:05.550974 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:05.546321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:52123)
0928 20:25:05.546471 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:05.546482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:05.546532 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:05.546686 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:05.546687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:05.546855 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:05.546904 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.546905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.547573 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.547574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.548114 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.548116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.548474 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.548475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.548514 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.548958 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:05.548963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:05.550858 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260928 20:25:05.551023 2027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:05.546413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52123 (local address 127.24.186.254:41579)
0928 20:25:05.546548 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:05.546550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:05.546558 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:05.546580 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:05.546581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:05.546592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:05.546656 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:05.546978 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.547503 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.548213 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.548406 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.548463 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.550948 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:52123: BlockingRecv error: recv got EOF from 127.24.186.193:52123 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:05.696751 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:05.691830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:39325)
0928 20:25:05.692042 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:05.692059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:05.692139 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:05.692366 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:05.692367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:05.692554 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:05.692600 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.692602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.693292 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.693294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.693832 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.693833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.694262 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.694263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.694326 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.694844 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:05.694849 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:05.696646 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":62,"threads_started":1}
W20260928 20:25:05.696902 2027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:05.692091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:39325 (local address 127.24.186.254:41579)
0928 20:25:05.692170 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:05.692172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:05.692181 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:05.692222 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:05.692223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:05.692235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:05.692317 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:05.692685 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.693203 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.693927 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.694139 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.694246 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.696738 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:39325: BlockingRecv error: recv got EOF from 127.24.186.194:39325 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:05 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:05.849700 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:05.844414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:44029)
0928 20:25:05.844635 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:05.844650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:05.844854 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:05.845057 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:05.845058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:05.845252 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:05.845306 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.845308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.845995 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.845996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.846550 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:05.846552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.846942 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.846943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:05.847006 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.847650 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:05.847655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:05.849589 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":113,"threads_started":1}
W20260928 20:25:05.849748 2027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:05.844550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:44029 (local address 127.24.186.254:41579)
0928 20:25:05.844640 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:05.844643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:05.844750 (+ 107us) server_negotiation.cc:407] Connection header received
0928 20:25:05.844959 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:05.844960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:05.844982 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:05.845051 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:05.845392 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.845907 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.846644 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:05.846843 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:05.846924 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:05.849671 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:44029: BlockingRecv error: recv got EOF from 127.24.186.195:44029 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:06.556617 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:06.551566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:51261)
0928 20:25:06.551734 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:06.551745 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:06.551849 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:06.551996 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:06.551996 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:06.552159 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:06.552205 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.552206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.552852 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.552854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.553388 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.553389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.553744 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.553745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.553805 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.554452 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:06.554458 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:06.556507 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260928 20:25:06.556676 2031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:06.551676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:51261 (local address 127.24.186.254:41579)
0928 20:25:06.551844 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:06.551846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:06.551865 (+ 19us) server_negotiation.cc:407] Connection header received
0928 20:25:06.551909 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:06.551909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:06.551920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:06.551980 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:06.552277 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.552777 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.553466 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.553664 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.553722 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.556602 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:51261: BlockingRecv error: recv got EOF from 127.24.186.193:51261 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:06.702677 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:06.697586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:53975)
0928 20:25:06.697766 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:06.697777 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:06.697826 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:06.697983 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:06.697984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:06.698144 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:06.698195 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.698197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.698974 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.698976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.699514 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.699516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.699951 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.699953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.700019 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.700473 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:06.700478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:06.702599 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
W20260928 20:25:06.702809 2031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:06.697692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:53975 (local address 127.24.186.254:41579)
0928 20:25:06.697801 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:06.697803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:06.697816 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:25:06.697903 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:06.697904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:06.697915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:06.697963 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:06.698265 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.698889 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.699622 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.699874 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.699945 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.702745 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:53975: BlockingRecv error: recv got EOF from 127.24.186.194:53975 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:06 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:06.855441 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:06.850377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:52489)
0928 20:25:06.850573 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:06.850584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:06.850656 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:06.850789 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:06.850790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:06.850954 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:06.851003 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.851004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.851742 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.851743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.852653 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:06.852656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.853012 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.853013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:06.853063 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.853498 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:06.853502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:06.855377 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":108,"threads_started":1}
W20260928 20:25:06.855624 2031 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:06.850521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:52489 (local address 127.24.186.254:41579)
0928 20:25:06.850576 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:06.850579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:06.850640 (+ 61us) server_negotiation.cc:407] Connection header received
0928 20:25:06.850709 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:06.850710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:06.850723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:06.850767 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:06.851119 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.851625 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.852723 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:06.852910 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:06.852966 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:06.855544 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:52489: BlockingRecv error: recv got EOF from 127.24.186.195:52489 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:07.562031 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:07.557332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:40295)
0928 20:25:07.557519 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:07.557530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:07.557580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:07.557755 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:07.557756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:07.557922 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:07.557970 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.557971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.558664 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.558665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.559200 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.559201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.559533 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.559533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.559571 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.560116 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:07.560121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:07.561959 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":116,"threads_started":1}
W20260928 20:25:07.562119 2035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:07.557451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:40295 (local address 127.24.186.254:41579)
0928 20:25:07.557602 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:07.557604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:07.557613 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:07.557641 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:07.557641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:07.557652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:07.557733 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:07.558057 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.558579 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.559275 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.559470 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.559520 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.562066 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:40295: BlockingRecv error: recv got EOF from 127.24.186.193:40295 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:07.708367 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:07.703428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:33569)
0928 20:25:07.703691 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:07.703710 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:07.703849 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:07.704033 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:07.704034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:07.704212 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:07.704257 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.704258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.704939 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.704940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.705463 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.705464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.705830 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.705831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.705893 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.706399 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:07.706404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:07.708305 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
W20260928 20:25:07.708479 2035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:07.703714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:33569 (local address 127.24.186.254:41579)
0928 20:25:07.703788 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:07.703790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:07.703827 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:25:07.703938 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:07.703939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:07.703953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:07.704028 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:07.704358 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.704852 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.705560 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.705757 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.705831 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.708402 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:33569: BlockingRecv error: recv got EOF from 127.24.186.194:33569 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:07 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:07.869154 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:07.856050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:56701)
0928 20:25:07.856262 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:07.856277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:07.856369 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:07.856529 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:07.856530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:07.856722 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:07.856768 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.856769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.857481 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.857482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.858113 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:07.858116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.858504 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.858505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:07.858579 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.867115 (+ 8536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:07.867122 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:07.869058 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
W20260928 20:25:07.869240 2035 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:07.856200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:56701 (local address 127.24.186.254:41579)
0928 20:25:07.856294 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:07.856296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:07.856348 (+ 52us) server_negotiation.cc:407] Connection header received
0928 20:25:07.856433 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:07.856434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:07.856449 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:07.856530 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:07.856854 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.857355 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.858219 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:07.858419 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:07.858505 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:07.869160 (+ 10655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:56701: BlockingRecv error: recv got EOF from 127.24.186.195:56701 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:08.568100 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:08.562709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:33575)
0928 20:25:08.562983 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:08.562998 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:08.563101 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:08.563327 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:08.563328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:08.563508 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:08.563572 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.563574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.564613 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.564614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.565263 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.565266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.565630 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.565631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.565683 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.566143 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:08.566148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:08.568038 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":79,"threads_started":1}
W20260928 20:25:08.568269 2039 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:08.562955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:33575 (local address 127.24.186.254:41579)
0928 20:25:08.563156 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:08.563157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:08.563168 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:25:08.563196 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:08.563197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:08.563208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:08.563279 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:08.563685 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.564541 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.565349 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.565562 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.565626 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.568149 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:33575: BlockingRecv error: recv got EOF from 127.24.186.193:33575 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:08.713995 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:08.708979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:40525)
0928 20:25:08.709138 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:08.709158 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:08.709216 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:08.709368 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:08.709369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:08.709527 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:08.709577 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.709578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.710235 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.710236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.711071 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.711075 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.711403 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.711404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.711442 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.711920 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:08.711925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:08.713933 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
W20260928 20:25:08.714159 2039 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:08.709177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:40525 (local address 127.24.186.254:41579)
0928 20:25:08.709230 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:08.709231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:08.709242 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:25:08.709275 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:08.709276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:08.709287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:08.709349 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:08.709646 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.710147 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.711148 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.711341 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.711394 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.714044 (+ 2650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:40525: BlockingRecv error: recv got EOF from 127.24.186.194:40525 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:08 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:08.874483 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:08.869749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:51051)
0928 20:25:08.869894 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:08.869908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:08.869981 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:08.870178 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:08.870179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:08.870360 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:08.870405 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.870407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.871069 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.871070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.871598 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:08.871599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.872050 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.872051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:08.872117 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.872628 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:08.872638 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:08.874423 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260928 20:25:08.874598 2039 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:08.869872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:51051 (local address 127.24.186.254:41579)
0928 20:25:08.869948 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:08.869950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:08.869959 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:08.870067 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:08.870068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:08.870083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:08.870167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:08.870486 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.870983 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.871705 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:08.871966 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:08.872050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:08.874522 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:51051: BlockingRecv error: recv got EOF from 127.24.186.195:51051 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:09.573530 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:09.568703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:35189)
0928 20:25:09.568860 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:09.568870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:09.568921 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:09.569194 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:09.569195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:09.569364 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:09.569415 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.569416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.570098 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.570099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.570638 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.570640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.571073 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.571074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.571120 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.571615 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:09.571619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:09.573467 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260928 20:25:09.573741 2043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:09.568876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:35189 (local address 127.24.186.254:41579)
0928 20:25:09.569007 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:09.569009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:09.569019 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:25:09.569054 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:09.569055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:09.569066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:09.569145 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:09.569495 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.569989 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.570757 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.570975 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.571036 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.573659 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:35189: BlockingRecv error: recv got EOF from 127.24.186.193:35189 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:09.719606 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:09.714563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:48255)
0928 20:25:09.714784 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:09.714798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:09.714878 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:09.715085 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:09.715086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:09.715271 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:09.715317 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.715319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.716093 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.716095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.716622 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.716624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.717003 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.717004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.717059 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.717571 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:09.717576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:09.719497 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":120,"threads_started":1}
W20260928 20:25:09.719653 2043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:09.714688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48255 (local address 127.24.186.254:41579)
0928 20:25:09.714789 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:09.714791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:09.714857 (+ 66us) server_negotiation.cc:407] Connection header received
0928 20:25:09.714949 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:09.714950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:09.714962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:09.715073 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:09.715404 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.715991 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.716710 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.716924 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.717005 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.719588 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:48255: BlockingRecv error: recv got EOF from 127.24.186.194:48255 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:09 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:09.880149 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:09.875044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:58451)
0928 20:25:09.875211 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:09.875222 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:09.875273 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:09.875420 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:09.875421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:09.875591 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:09.875637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.875639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.876366 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.876367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.877014 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:09.877016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.877419 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.877421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:09.877468 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.877970 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:09.877974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:09.880037 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
W20260928 20:25:09.880220 2043 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:09.875173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:58451 (local address 127.24.186.254:41579)
0928 20:25:09.875269 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:09.875270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:09.875282 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:25:09.875324 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:09.875325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:09.875337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:09.875401 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:09.875749 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.876280 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.877124 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:09.877314 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:09.877381 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:09.880147 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:58451: BlockingRecv error: recv got EOF from 127.24.186.195:58451 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:10.579157 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:10.574114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:51625)
0928 20:25:10.574258 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:10.574270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:10.574333 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:10.574596 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:10.574597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:10.574762 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:10.574812 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.574813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.575509 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.575510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.576074 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.576075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.576435 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.576436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.576476 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.576942 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:10.576947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:10.579076 (+ 2129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
W20260928 20:25:10.579371 2047 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:10.574301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:51625 (local address 127.24.186.254:41579)
0928 20:25:10.574455 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:10.574456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:10.574464 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:10.574490 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:10.574491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:10.574500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:10.574580 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:10.574896 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.575425 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.576165 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.576363 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.576428 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.579291 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:51625: BlockingRecv error: recv got EOF from 127.24.186.193:51625 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:10.725052 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:10.720232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:48935)
0928 20:25:10.720391 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:10.720404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:10.720470 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:10.720686 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:10.720687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:10.720899 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:10.720962 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.720965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.721796 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.721797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.722346 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.722348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.722689 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.722690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.722730 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.723198 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:10.723202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:10.724987 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
W20260928 20:25:10.725238 2047 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:10.720371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48935 (local address 127.24.186.254:41579)
0928 20:25:10.720423 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:10.720425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:10.720460 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:25:10.720552 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:10.720553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:10.720565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:10.720663 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:10.721051 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.721699 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.722434 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.722621 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.722676 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.725163 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:48935: BlockingRecv error: recv got EOF from 127.24.186.194:48935 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:10 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:10.886340 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:10.880863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:49897)
0928 20:25:10.881092 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:10.881116 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:10.881216 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:10.881407 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:10.881408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:10.881577 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:10.881630 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.881632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.882443 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.882444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.882968 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:10.882970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.883357 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.883358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:10.883417 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.884026 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:10.884031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:10.886238 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":163,"threads_started":1}
W20260928 20:25:10.886445 2047 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:10.880978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:49897 (local address 127.24.186.254:41579)
0928 20:25:10.881045 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:10.881073 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:10.881212 (+ 139us) server_negotiation.cc:407] Connection header received
0928 20:25:10.881271 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:10.881272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:10.881284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:10.881347 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:10.881795 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.882339 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.883062 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:10.883265 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:10.883357 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:10.886373 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:49897: BlockingRecv error: recv got EOF from 127.24.186.195:49897 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:11.584997 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:11.579719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:38641)
0928 20:25:11.579876 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:11.579888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:11.579952 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:11.580303 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:11.580304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:11.580527 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:11.580581 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.580583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.581238 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.581238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.581780 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.581782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.582175 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.582176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.582223 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.582665 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:11.582670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:11.584909 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
W20260928 20:25:11.585260 2051 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:11.580025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:38641 (local address 127.24.186.254:41579)
0928 20:25:11.580142 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:11.580144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:11.580151 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:25:11.580184 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:11.580185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:11.580205 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:11.580282 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:11.580647 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.581154 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.581876 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.582093 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.582159 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.585167 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:38641: BlockingRecv error: recv got EOF from 127.24.186.193:38641 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:11.731060 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:11.725729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:46185)
0928 20:25:11.725906 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:11.725922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:11.726004 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:11.726179 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:11.726180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:11.726381 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:11.726445 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.726447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.727331 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.727333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.727998 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.728001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.728447 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.728449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.728501 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.728983 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:11.728988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:11.730983 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
W20260928 20:25:11.731194 2051 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:11.725925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:46185 (local address 127.24.186.254:41579)
0928 20:25:11.726009 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:11.726011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:11.726060 (+ 49us) server_negotiation.cc:407] Connection header received
0928 20:25:11.726091 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:11.726092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:11.726108 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:11.726178 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:11.726579 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.727218 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.728132 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.728353 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.728413 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.731104 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:46185: BlockingRecv error: recv got EOF from 127.24.186.194:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:11 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:11.892109 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:11.886795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:55885)
0928 20:25:11.886965 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:11.886980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:11.887114 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:11.887295 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:11.887296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:11.887488 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:11.887545 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.887547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.888273 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.888274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.888794 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:11.888795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.889182 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.889183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:11.889239 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.889688 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:11.889692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:11.891994 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20260928 20:25:11.892175 2051 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:11.886891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:55885 (local address 127.24.186.254:41579)
0928 20:25:11.886970 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:11.886972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:11.887094 (+ 122us) server_negotiation.cc:407] Connection header received
0928 20:25:11.887195 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:11.887196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:11.887215 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:11.887286 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:11.887621 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.888185 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.888891 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:11.889084 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:11.889183 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:11.892109 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:55885: BlockingRecv error: recv got EOF from 127.24.186.195:55885 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:12.590504 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:12.585610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:48539)
0928 20:25:12.585804 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:12.585818 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:12.585893 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:12.586098 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:12.586099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:12.586263 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:12.586342 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.586345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.587039 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.587040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.587570 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.587572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.587941 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.587942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.587983 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.588527 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:12.588537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:12.590444 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
W20260928 20:25:12.590965 2055 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:12.585714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48539 (local address 127.24.186.254:41579)
0928 20:25:12.585868 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:12.585871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:12.585883 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:25:12.585983 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:12.585983 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:12.585995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:12.586079 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:12.586427 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.586957 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.587661 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.587877 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.587930 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.590774 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:48539: BlockingRecv error: recv got EOF from 127.24.186.193:48539 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:12.737712 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:12.731927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:46903)
0928 20:25:12.732103 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:12.732123 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:12.732194 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:12.732334 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:12.732335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:12.732507 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:12.732560 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.732571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.733303 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.733304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.733834 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.733836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.734237 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.734238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.734283 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.734846 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:12.734852 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:12.737593 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":107,"threads_started":1}
W20260928 20:25:12.737973 2055 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:12.731978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:46903 (local address 127.24.186.254:41579)
0928 20:25:12.732064 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:12.732067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:12.732196 (+ 129us) server_negotiation.cc:407] Connection header received
0928 20:25:12.732247 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:12.732248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:12.732259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:12.732311 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:12.732668 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.733200 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.733942 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.734140 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.734205 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.737754 (+ 3549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:46903: BlockingRecv error: recv got EOF from 127.24.186.194:46903 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:12 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:12.903275 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:12.892690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:41153)
0928 20:25:12.892875 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:12.892890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:12.892958 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:12.893187 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:12.893188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:12.893371 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:12.893418 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.893420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.894116 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.894117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.894632 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:12.894634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.895013 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.895014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:12.895081 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.900912 (+ 5831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:12.900918 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:12.903213 (+ 2295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
W20260928 20:25:12.903468 2055 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:12.892968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:41153 (local address 127.24.186.254:41579)
0928 20:25:12.893055 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:12.893057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:12.893064 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:25:12.893088 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:12.893094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:12.893106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:12.893178 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:12.893512 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.894020 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.894726 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:12.894932 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:12.895016 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:12.903360 (+ 8344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:41153: BlockingRecv error: recv got EOF from 127.24.186.195:41153 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:13.596405 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:13.591029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:50669)
0928 20:25:13.591172 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:13.591187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:13.591269 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:13.591421 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:13.591421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:13.591591 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:13.591648 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.591650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.592677 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.592678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.593282 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.593284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.593784 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.593785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.593834 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.594383 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:13.594388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:13.596329 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":53,"threads_started":1}
W20260928 20:25:13.596612 2059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:13.591147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:50669 (local address 127.24.186.254:41579)
0928 20:25:13.591292 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:13.591294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:13.591302 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:13.591324 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:13.591325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:13.591335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:13.591395 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:13.591720 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.592450 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.593440 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.593677 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.593762 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.596473 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:50669: BlockingRecv error: recv got EOF from 127.24.186.193:50669 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:13.744115 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:13.738406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:48955)
0928 20:25:13.738611 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:13.738624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:13.738693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:13.738895 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:13.738897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:13.739129 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:13.739205 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.739207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.740370 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.740371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.740948 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.740950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.741379 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.741380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.741418 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.741915 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:13.741920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:13.744004 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":117,"threads_started":1}
W20260928 20:25:13.744391 2059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:13.738563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48955 (local address 127.24.186.254:41579)
0928 20:25:13.738619 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:13.738621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:13.738676 (+ 55us) server_negotiation.cc:407] Connection header received
0928 20:25:13.738788 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:13.738790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:13.738805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:13.738865 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:13.739374 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.740270 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.741064 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.741306 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.741363 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.744316 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:48955: BlockingRecv error: recv got EOF from 127.24.186.194:48955 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:13 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:13.908751 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:13.903922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:45721)
0928 20:25:13.904129 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:13.904142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:13.904205 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:13.904347 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:13.904347 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:13.904530 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:13.904579 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.904580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.905286 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.905287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.905853 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:13.905855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.906227 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.906229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:13.906282 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.906782 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:13.906787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:13.908682 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":145,"threads_started":1}
W20260928 20:25:13.908962 2059 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:13.904041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:45721 (local address 127.24.186.254:41579)
0928 20:25:13.904142 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:13.904143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:13.904191 (+ 48us) server_negotiation.cc:407] Connection header received
0928 20:25:13.904263 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:13.904264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:13.904278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:13.904321 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:13.904674 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.905200 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.905935 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:13.906120 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:13.906180 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:13.908885 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:45721: BlockingRecv error: recv got EOF from 127.24.186.195:45721 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:14.602222 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:14.597050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:51153)
0928 20:25:14.597239 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:14.597253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:14.597334 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:14.597475 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:14.597475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:14.597654 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:14.597703 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.597705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.598391 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.598392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.598924 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.598926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.599270 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.599271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.599312 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.599779 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:14.599783 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:14.602063 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
W20260928 20:25:14.602252 2063 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:14.597197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:51153 (local address 127.24.186.254:41579)
0928 20:25:14.597336 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:14.597338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:14.597347 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:14.597388 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:14.597389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:14.597400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:14.597452 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:14.597776 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.598291 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.599007 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.599201 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.599253 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.602197 (+ 2944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:51153: BlockingRecv error: recv got EOF from 127.24.186.193:51153 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:14.750056 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:14.744690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:35263)
0928 20:25:14.744850 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:14.744867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:14.744920 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:14.745381 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:14.745382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:14.745687 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:14.745745 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.745753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.746457 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.746458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.746990 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.746992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.747364 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.747365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.747420 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.748065 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:14.748071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:14.749948 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260928 20:25:14.750097 2063 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:14.745089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:35263 (local address 127.24.186.254:41579)
0928 20:25:14.745210 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:14.745212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:14.745222 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:25:14.745252 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:14.745252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:14.745265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:14.745370 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:14.745858 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.746363 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.747087 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.747283 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.747367 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.750033 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:35263: BlockingRecv error: recv got EOF from 127.24.186.194:35263 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:14 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:14.914264 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:14.909270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:55805)
0928 20:25:14.909436 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:14.909453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:14.909503 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:14.909651 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:14.909652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:14.909837 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:14.909883 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.909885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.910693 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.910694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.911263 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:14.911265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.911646 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.911646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:14.911688 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.912294 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:14.912304 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:14.914154 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
W20260928 20:25:14.914305 2063 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:14.909367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:55805 (local address 127.24.186.254:41579)
0928 20:25:14.909484 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:14.909486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:14.909499 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:25:14.909552 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:14.909553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:14.909566 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:14.909623 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:14.910046 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.910589 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.911377 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:14.911567 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:14.911628 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:14.914244 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:55805: BlockingRecv error: recv got EOF from 127.24.186.195:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:15.608040 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:15.602954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:53573)
0928 20:25:15.603108 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:15.603118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:15.603177 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:15.603434 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:15.603435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:15.603642 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:15.603690 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.603691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.604449 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.604450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.605001 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.605003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.605388 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.605389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.605445 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.605968 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:15.605972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:15.607967 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
W20260928 20:25:15.608211 2067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:15.603047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:53573 (local address 127.24.186.254:41579)
0928 20:25:15.603282 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:15.603284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:15.603292 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:15.603316 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:15.603317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:15.603328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:15.603413 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:15.603788 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.604342 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.605114 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.605311 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.605390 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.608111 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:53573: BlockingRecv error: recv got EOF from 127.24.186.193:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":82,"threads_started":1}
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:15.755548 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:15.750674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:53013)
0928 20:25:15.750885 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:15.750899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:15.750968 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:15.751181 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:15.751182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:15.751348 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:15.751397 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.751398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.752062 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.752063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.752597 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.752599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.752930 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.752931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.752975 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.753470 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:15.753474 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:15.755487 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
W20260928 20:25:15.756021 2067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:15.750933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:53013 (local address 127.24.186.254:41579)
0928 20:25:15.751018 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:15.751021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:15.751031 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:25:15.751064 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:15.751064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:15.751077 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:15.751131 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:15.751469 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.751991 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.752674 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.752859 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.752908 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.755973 (+ 3065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:53013: BlockingRecv error: recv got EOF from 127.24.186.194:53013 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:15 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:15.919612 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:15.914914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:38615)
0928 20:25:15.915086 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:15.915097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:15.915180 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:15.915367 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:15.915368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:15.915558 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:15.915602 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.915604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.916360 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.916361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.916894 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:15.916896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.917283 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.917284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:15.917342 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.917790 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:15.917794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:15.919495 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
W20260928 20:25:15.919641 2067 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:15.915012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:38615 (local address 127.24.186.254:41579)
0928 20:25:15.915068 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:15.915070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:15.915171 (+ 101us) server_negotiation.cc:407] Connection header received
0928 20:25:15.915264 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:15.915265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:15.915280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:15.915361 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:15.915703 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.916274 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.916996 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:15.917192 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:15.917283 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:15.919573 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:38615: BlockingRecv error: recv got EOF from 127.24.186.195:38615 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:16.614120 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:16.608677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:54233)
0928 20:25:16.608858 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:16.608871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:16.608998 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:16.609229 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:16.609230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:16.609479 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:16.609539 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.609541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.610299 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.610300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.610819 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.610820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.611190 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.611191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.611232 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.611776 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:16.611781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:16.614050 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
W20260928 20:25:16.614310 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:16.608836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:54233 (local address 127.24.186.254:41579)
0928 20:25:16.608999 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:16.609001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:16.609012 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:25:16.609099 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:16.609101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:16.609117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:16.609194 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:16.609636 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.610184 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.610919 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.611115 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.611172 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.614190 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:54233: BlockingRecv error: recv got EOF from 127.24.186.193:54233 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:16.762274 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:16.756326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:48665)
0928 20:25:16.756500 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:16.756514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:16.756577 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:16.756708 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:16.756709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:16.756886 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:16.756933 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.756935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.757633 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.757634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.758225 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.758227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.758654 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.758655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.758718 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.759354 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:16.759358 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:16.762108 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
W20260928 20:25:16.762338 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:16.756468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48665 (local address 127.24.186.254:41579)
0928 20:25:16.756548 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:16.756550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:16.756586 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:25:16.756633 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:16.756634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:16.756645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:16.756700 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:16.757039 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.757531 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.758345 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.758536 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.758601 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.762248 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:48665: BlockingRecv error: recv got EOF from 127.24.186.194:48665 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:16 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.195@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:16.925333 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:16.920228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.195:53627)
0928 20:25:16.920352 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:16.920393 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:16.920507 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:16.920747 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:16.920748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:16.920994 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:16.921048 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.921050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.921688 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.921688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.922244 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:16.922245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.922568 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.922568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:16.922605 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.923109 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:16.923114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:16.925273 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
W20260928 20:25:16.925486 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:16.920286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:53627 (local address 127.24.186.254:41579)
0928 20:25:16.920355 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:16.920357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:16.920499 (+ 142us) server_negotiation.cc:407] Connection header received
0928 20:25:16.920624 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:16.920625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:16.920640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:16.920719 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:16.921119 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.921610 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.922323 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:16.922498 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:16.922547 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:16.925394 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.195:53627: BlockingRecv error: recv got EOF from 127.24.186.195:53627 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 20:25:17 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:17 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.193@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:17.619665 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:17.614806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.193:60519)
0928 20:25:17.615015 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:17.615027 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:17.615086 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:17.615209 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:17.615209 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:17.615372 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:17.615420 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:17.615422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.616226 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.616227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:17.616778 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:17.616780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.617189 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.617191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:17.617232 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:17.617732 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:17.617736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:17.619602 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":120,"threads_started":1}
W20260928 20:25:17.619864 2075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:17.614954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:60519 (local address 127.24.186.254:41579)
0928 20:25:17.615062 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:17.615065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:17.615074 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:17.615135 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:17.615136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:17.615146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:17.615198 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:17.615486 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.616150 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.616881 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.617086 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.617152 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:17.619718 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.193:60519: BlockingRecv error: recv got EOF from 127.24.186.193:60519 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
Sep 28 20:25:17 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 20:25:17 dist-test-slave-k477 krb5kdc[1016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.24.186.194@KRBTEST.COM for oryx/127.24.186.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 20:25:17.768031 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:17.763155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:41579 (local address 127.24.186.194:44727)
0928 20:25:17.763349 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:17.763364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:17.763434 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:17.763602 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:17.763602 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:17.763767 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:17.763888 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:17.763891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.764561 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.764562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:17.765079 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:17.765080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.765443 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.765444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:17.765488 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:17.765980 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:17.765985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:17.767931 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.24.186.254:41579: Server oryx/127.24.186.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
W20260928 20:25:17.768131 2075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:17.763321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:44727 (local address 127.24.186.254:41579)
0928 20:25:17.763414 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:17.763416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:17.763428 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:25:17.763502 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:17.763503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:17.763516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:17.763581 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:17.763976 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.764482 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.765180 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:17.765365 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:17.765421 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:17.768058 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.24.186.194:44727: BlockingRecv error: recv got EOF from 127.24.186.194:44727 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 20:25:17.827003 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1599
I20260928 20:25:17.834174 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1751
I20260928 20:25:17.840510 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1885
I20260928 20:25:17.847270 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 1529
2026-09-28T20:25:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17645 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T20:25:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:17Z Disabled control of system clock
I20260928 20:25:17.880671 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:37011
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:37011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 20:25:17.968888 2080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:17.969131 2080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:17.969153 2080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:17.970525 2080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:17.970587 2080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:17.970602 2080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:17.970616 2080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:17.970629 2080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:17.972090 2080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:37011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:17.972304 2080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:17.972559 2080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:25:17.978915 2080 server_base.cc:1034] running on GCE node
W20260928 20:25:17.978886 2085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:17.978897 2088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:17.978881 2086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:17.979455 2080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:17.979768 2080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:17.981086 2080 hybrid_clock.cc:648] HybridClock initialized: now 1790627117981063 us; error 42 us; skew 500 ppm
I20260928 20:25:17.988679 2080 webserver.cc:460] Webserver started at http://127.24.186.254:38163/ using document root <none> and password file <none>
I20260928 20:25:17.988978 2080 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:17.989032 2080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:17.989207 2080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:17.990352 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "9b27eee1de2441e499e9e9213b29729d"
format_stamp: "Formatted at 2026-09-28 20:25:17 on dist-test-slave-k477"
server_key: "7fc4506ce702fa7565e7cd4dd999ce86"
server_key_iv: "64fde46ddf2be14b68aa22780f85a9c2"
server_key_version: "encryptionkey@0"
I20260928 20:25:17.990739 2080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "9b27eee1de2441e499e9e9213b29729d"
format_stamp: "Formatted at 2026-09-28 20:25:17 on dist-test-slave-k477"
server_key: "7fc4506ce702fa7565e7cd4dd999ce86"
server_key_iv: "64fde46ddf2be14b68aa22780f85a9c2"
server_key_version: "encryptionkey@0"
I20260928 20:25:17.992182 2080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:25:17.993009 2094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:17.993321 2080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:17.993433 2080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9b27eee1de2441e499e9e9213b29729d"
format_stamp: "Formatted at 2026-09-28 20:25:17 on dist-test-slave-k477"
server_key: "7fc4506ce702fa7565e7cd4dd999ce86"
server_key_iv: "64fde46ddf2be14b68aa22780f85a9c2"
server_key_version: "encryptionkey@0"
I20260928 20:25:17.993512 2080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.005012 2080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.005336 2080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.005494 2080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.009653 2080 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:37011
I20260928 20:25:18.009690 2146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:37011 every 8 connection(s)
I20260928 20:25:18.010069 2080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:25:18.010733 2147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:18.013456 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d: Bootstrap starting.
I20260928 20:25:18.014060 2147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:18.014348 2147 log.cc:824] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:18.015043 2147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d: No bootstrap required, opened a new log
I20260928 20:25:18.016702 2147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } }
I20260928 20:25:18.016853 2147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:18.016877 2147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b27eee1de2441e499e9e9213b29729d, State: Initialized, Role: FOLLOWER
I20260928 20:25:18.017031 2147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } }
I20260928 20:25:18.017099 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2080
I20260928 20:25:18.017141 2147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:25:18.017180 2147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:25:18.017220 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:25:18.017231 2147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:18.017409 25323 external_mini_cluster.cc:1428] Setting key 55ee7a46cd28d05f4fcde767f3b3e4ac
I20260928 20:25:18.017999 2147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } }
I20260928 20:25:18.018138 2147 leader_election.cc:302] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b27eee1de2441e499e9e9213b29729d; no voters:
I20260928 20:25:18.018290 2147 leader_election.cc:288] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:25:18.018378 2152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:25:18.018530 2147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:25:18.018541 2152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [term 1 LEADER]: Becoming Leader. State: Replica: 9b27eee1de2441e499e9e9213b29729d, State: Running, Role: LEADER
I20260928 20:25:18.018839 2152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } }
I20260928 20:25:18.019261 2154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b27eee1de2441e499e9e9213b29729d. Latest consensus state: current_term: 1 leader_uuid: "9b27eee1de2441e499e9e9213b29729d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } } }
I20260928 20:25:18.019337 2154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:18.019649 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b27eee1de2441e499e9e9213b29729d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b27eee1de2441e499e9e9213b29729d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37011 } } }
I20260928 20:25:18.019711 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:18.019963 2161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:25:18.020474 2161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:25:18.021957 2161 catalog_manager.cc:1347] Generated new cluster ID: df6f78edac4c41c6811cbb286dbe589a
I20260928 20:25:18.022028 2161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:25:18.024008 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.017932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.24.186.254:37011)
0928 20:25:18.018171 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.018181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.018213 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:25:18.018334 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.018343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.018417 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.018813 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.018914 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.020914 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.021847 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.022454 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.022567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.023306 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.023310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.023310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.023506 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.023545 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.023559 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.023624 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.023720 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.023792 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":88,"threads_started":1}
I20260928 20:25:18.040047 2161 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:25:18.040800 2161 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:25:18.044610 2161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b27eee1de2441e499e9e9213b29729d: Generated new TSK 0
I20260928 20:25:18.044797 2161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:25:18.049511 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--builtin_ntp_servers=127.24.186.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 20:25:18.132229 2171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.132485 2171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.132524 2171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.133749 2171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.133826 2171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.133872 2171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:25:18.135314 2171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.135609 2171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.135963 2171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.141116 2176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.141179 2179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.141462 2171 server_base.cc:1034] running on GCE node
W20260928 20:25:18.141464 2177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.141746 2171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.142022 2171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.143215 2171 hybrid_clock.cc:648] HybridClock initialized: now 1790627118143174 us; error 55 us; skew 500 ppm
I20260928 20:25:18.144629 2171 webserver.cc:460] Webserver started at http://127.24.186.193:44349/ using document root <none> and password file <none>
I20260928 20:25:18.144896 2171 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.144971 2171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.145121 2171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.146203 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "ea27dbaf93ab4bbdb4b3356457aeb98b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "906ffa24aa7228eeb02d43af1dfe691a"
server_key_iv: "8a34844232f65c829ec03775abcb1115"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.146569 2171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "ea27dbaf93ab4bbdb4b3356457aeb98b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "906ffa24aa7228eeb02d43af1dfe691a"
server_key_iv: "8a34844232f65c829ec03775abcb1115"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.147970 2171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:25:18.148904 2185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.149139 2171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.149240 2171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "ea27dbaf93ab4bbdb4b3356457aeb98b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "906ffa24aa7228eeb02d43af1dfe691a"
server_key_iv: "8a34844232f65c829ec03775abcb1115"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.149333 2171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.158157 2171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.158478 2171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.158643 2171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.158890 2171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:18.159215 2171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:18.159277 2171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.159320 2171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:18.159343 2171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.165170 2171 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:36471
I20260928 20:25:18.165211 2298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:36471 every 8 connection(s)
I20260928 20:25:18.165577 2171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:25:18.170984 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.166411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:51703 (local address 127.24.186.254:37011)
0928 20:25:18.166492 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.166495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.166740 (+ 245us) server_negotiation.cc:407] Connection header received
0928 20:25:18.167255 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.167256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.167269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.167328 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.168025 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.168557 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.169529 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.169724 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.169790 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.170648 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.170650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.170650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.170659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.170674 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.170678 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.170737 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.170872 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.170903 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:25:18.171216 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.166381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:37011 (local address 127.24.186.193:51703)
0928 20:25:18.166611 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:18.166646 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:18.167130 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:18.167408 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:18.167411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:18.167420 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:18.167926 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.167930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.168655 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.168656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.169433 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.169435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.169831 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.169832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.169954 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.170473 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:18.170474 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:18.170510 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:18.170533 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:18.170779 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:18.170785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:18.170787 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:25:18.170858 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:25:18.170947 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":103,"threads_started":1}
I20260928 20:25:18.171902 2299 heartbeater.cc:347] Connected to a master server at 127.24.186.254:37011
I20260928 20:25:18.172031 2299 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:18.172222 2299 heartbeater.cc:511] Master 127.24.186.254:37011 requested a full tablet report, sending...
I20260928 20:25:18.172730 2111 ts_manager.cc:194] Registered new tserver with Master: ea27dbaf93ab4bbdb4b3356457aeb98b (127.24.186.193:36471)
I20260928 20:25:18.173309 2111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.193:51703
I20260928 20:25:18.174775 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2171
I20260928 20:25:18.174850 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:25:18.175000 25323 external_mini_cluster.cc:1428] Setting key ba45d00e805802c49a07698537d44330
I20260928 20:25:18.175951 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--builtin_ntp_servers=127.24.186.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 20:25:18.259984 2302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.260248 2302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.260289 2302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.261510 2302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.261587 2302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.261634 2302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:25:18.263633 2302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.263965 2302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.264319 2302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.269986 2310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.270022 2308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.270119 2302 server_base.cc:1034] running on GCE node
W20260928 20:25:18.269999 2307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.270480 2302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.270825 2302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.272032 2302 hybrid_clock.cc:648] HybridClock initialized: now 1790627118272006 us; error 45 us; skew 500 ppm
I20260928 20:25:18.274003 2302 webserver.cc:460] Webserver started at http://127.24.186.194:45787/ using document root <none> and password file <none>
I20260928 20:25:18.274482 2302 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.274618 2302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.274885 2302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.276172 2302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "ba06053310944b118a6667ed1a6cef9c"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4a9898b8d8b4e25c71857c25c3373e7c"
server_key_iv: "7e922cdfe8cd116d1eee605ed17ae860"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.276562 2302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "ba06053310944b118a6667ed1a6cef9c"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4a9898b8d8b4e25c71857c25c3373e7c"
server_key_iv: "7e922cdfe8cd116d1eee605ed17ae860"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.278190 2302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:25:18.279731 2316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.280174 2302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.280329 2302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "ba06053310944b118a6667ed1a6cef9c"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4a9898b8d8b4e25c71857c25c3373e7c"
server_key_iv: "7e922cdfe8cd116d1eee605ed17ae860"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.280438 2302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.300287 2302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.300642 2302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.300832 2302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.301092 2302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:18.301448 2302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:18.302433 2302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260928 20:25:18.302587 2302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:18.302623 2302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.310527 2302 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:46427
I20260928 20:25:18.310632 2429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:46427 every 8 connection(s)
I20260928 20:25:18.310981 2302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:25:18.311399 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2302
I20260928 20:25:18.311504 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:25:18.311743 25323 external_mini_cluster.cc:1428] Setting key 60b2b292f29ec8765baf560fe91d1456
I20260928 20:25:18.312861 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--builtin_ntp_servers=127.24.186.212:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 20:25:18.317153 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.311872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:51305 (local address 127.24.186.254:37011)
0928 20:25:18.311948 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.311951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.312193 (+ 242us) server_negotiation.cc:407] Connection header received
0928 20:25:18.312856 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.312857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.312869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.312928 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.313649 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.314247 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.315245 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.315575 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.315652 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.316827 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.316829 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.316829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.316836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.316855 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.316860 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.316926 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.316994 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.317028 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:25:18.317329 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.311875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:37011 (local address 127.24.186.194:51305)
0928 20:25:18.312075 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:18.312118 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:18.312741 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:18.313008 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:18.313012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:18.313020 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:18.313538 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.313542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.314371 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.314372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.315172 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.315174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.315700 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.315701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.315922 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.316651 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:18.316651 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:18.316682 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:18.316707 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:18.316933 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:18.316937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:18.316939 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:25:18.316993 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 20:25:18.317094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260928 20:25:18.317909 2430 heartbeater.cc:347] Connected to a master server at 127.24.186.254:37011
I20260928 20:25:18.318024 2430 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:18.318243 2430 heartbeater.cc:511] Master 127.24.186.254:37011 requested a full tablet report, sending...
I20260928 20:25:18.318717 2111 ts_manager.cc:194] Registered new tserver with Master: ba06053310944b118a6667ed1a6cef9c (127.24.186.194:46427)
I20260928 20:25:18.319167 2111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.194:51305
W20260928 20:25:18.401430 2433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.401652 2433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.401676 2433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.402957 2433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.403023 2433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.403053 2433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:25:18.404557 2433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:37011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.404799 2433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.405210 2433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.410631 2438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.410632 2439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.410753 2441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.410773 2433 server_base.cc:1034] running on GCE node
I20260928 20:25:18.411132 2433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.411408 2433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.412616 2433 hybrid_clock.cc:648] HybridClock initialized: now 1790627118412563 us; error 65 us; skew 500 ppm
I20260928 20:25:18.414345 2433 webserver.cc:460] Webserver started at http://127.24.186.195:34405/ using document root <none> and password file <none>
I20260928 20:25:18.414635 2433 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.414736 2433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.414893 2433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.416153 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "231d5b7a63f14665a5d7d00ad6a7905b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "1c3187a0deab7f1571f4930a4603b7f2"
server_key_iv: "3eaef0deeb997bd6e398048854df2669"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.416550 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "231d5b7a63f14665a5d7d00ad6a7905b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "1c3187a0deab7f1571f4930a4603b7f2"
server_key_iv: "3eaef0deeb997bd6e398048854df2669"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.417948 2433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:25:18.418869 2447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.419072 2433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.419214 2433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "231d5b7a63f14665a5d7d00ad6a7905b"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "1c3187a0deab7f1571f4930a4603b7f2"
server_key_iv: "3eaef0deeb997bd6e398048854df2669"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.419317 2433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.453840 2433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.454216 2433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.454386 2433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.454649 2433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:18.455053 2433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:18.455127 2433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.455164 2433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:18.455191 2433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.461681 2433 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:39397
I20260928 20:25:18.461719 2560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:39397 every 8 connection(s)
I20260928 20:25:18.462122 2433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:25:18.467985 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.462936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:36965 (local address 127.24.186.254:37011)
0928 20:25:18.463036 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.463039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.463286 (+ 247us) server_negotiation.cc:407] Connection header received
0928 20:25:18.463897 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.463898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.463912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.463959 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.464689 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.465224 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.466418 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.466626 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.466715 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.467639 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.467640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.467641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.467646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.467662 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.467667 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.467740 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.467864 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.467897 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:25:18.468219 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.462888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:37011 (local address 127.24.186.195:36965)
0928 20:25:18.463137 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:18.463186 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:18.463746 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:18.464030 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:18.464034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:18.464045 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:18.464572 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.464577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.465341 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.465342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.466309 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:18.466311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:18.466720 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:18.466721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:18.466862 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:18.467466 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:18.467466 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:18.467495 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:18.467519 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:18.467745 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:18.467750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:18.467753 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:25:18.467866 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 20:25:18.467974 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260928 20:25:18.469017 2561 heartbeater.cc:347] Connected to a master server at 127.24.186.254:37011
I20260928 20:25:18.469137 2561 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:18.469448 2561 heartbeater.cc:511] Master 127.24.186.254:37011 requested a full tablet report, sending...
I20260928 20:25:18.469676 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2433
I20260928 20:25:18.469790 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:25:18.469969 2111 ts_manager.cc:194] Registered new tserver with Master: 231d5b7a63f14665a5d7d00ad6a7905b (127.24.186.195:39397)
I20260928 20:25:18.470026 25323 external_mini_cluster.cc:1428] Setting key 361bad8af481553f5bdeb9206c299dd8
I20260928 20:25:18.470417 2111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.195:36965
I20260928 20:25:18.471313 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:25:18.473608 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2171
W20260928 20:25:18.473614 2151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:18.473021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.24.186.254:37011)
0928 20:25:18.473105 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.473107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.473117 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:25:18.473206 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.473207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.473219 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.473274 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.473493 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56892: BlockingRecv error: recv got EOF from 127.0.0.1:56892 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 20:25:18.479368 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2302
I20260928 20:25:18.483950 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2433
I20260928 20:25:18.488266 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2080
2026-09-28T20:25:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (636 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T20:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:18Z Disabled control of system clock
I20260928 20:25:18.517091 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34033
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:25:18.606451 2572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.606665 2572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.606688 2572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.607793 2572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:18.607863 2572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.607877 2572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.607889 2572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:18.607903 2572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:18.609234 2572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:34033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.609467 2572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.610010 2572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.615128 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.615166 2578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.615187 2577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.615243 2572 server_base.cc:1034] running on GCE node
I20260928 20:25:18.615576 2572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.615953 2572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.617179 2572 hybrid_clock.cc:648] HybridClock initialized: now 1790627118617160 us; error 36 us; skew 500 ppm
I20260928 20:25:18.618754 2572 webserver.cc:460] Webserver started at http://127.24.186.254:46039/ using document root <none> and password file <none>
I20260928 20:25:18.619032 2572 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.619136 2572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.619311 2572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.620536 2572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "8617b4db31fe4bc3891e3c9cbbc35324"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "dbca70da1a92dc45f5cf96e6c2bf9e45"
server_key_iv: "c67c62a3a7ab30b26c0af3866b9140ce"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.620930 2572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "8617b4db31fe4bc3891e3c9cbbc35324"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "dbca70da1a92dc45f5cf96e6c2bf9e45"
server_key_iv: "c67c62a3a7ab30b26c0af3866b9140ce"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.622383 2572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:25:18.623224 2586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.623437 2572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.623551 2572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "8617b4db31fe4bc3891e3c9cbbc35324"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "dbca70da1a92dc45f5cf96e6c2bf9e45"
server_key_iv: "c67c62a3a7ab30b26c0af3866b9140ce"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.623646 2572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.627205 2572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.627518 2572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.627686 2572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.633513 2572 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:34033
I20260928 20:25:18.633580 2638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:34033 every 8 connection(s)
I20260928 20:25:18.633956 2572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:25:18.634732 2639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:18.637715 2639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324: Bootstrap starting.
I20260928 20:25:18.639230 2639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:18.639990 2639 log.cc:824] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:18.641000 2639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324: No bootstrap required, opened a new log
I20260928 20:25:18.641975 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2572
I20260928 20:25:18.642124 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:25:18.642339 25323 external_mini_cluster.cc:1428] Setting key f1e05af030b8f66fdfe5bccce895b46f
I20260928 20:25:18.643512 2639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } }
I20260928 20:25:18.643707 2639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:18.643747 2639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8617b4db31fe4bc3891e3c9cbbc35324, State: Initialized, Role: FOLLOWER
I20260928 20:25:18.643939 2639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } }
I20260928 20:25:18.644059 2639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:25:18.644106 2639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:25:18.644155 2639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:18.644999 2639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } }
I20260928 20:25:18.645138 2639 leader_election.cc:302] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8617b4db31fe4bc3891e3c9cbbc35324; no voters:
I20260928 20:25:18.645162 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.643029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.24.186.254:34033)
0928 20:25:18.643283 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.643293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.643332 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:25:18.643456 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.643463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.643525 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.643684 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.644552 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.644557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.644557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.644562 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.644611 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.644625 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.644717 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.644846 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.644908 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260928 20:25:18.645309 2639 leader_election.cc:288] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:25:18.645582 2644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:25:18.645640 2639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:25:18.645973 2644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [term 1 LEADER]: Becoming Leader. State: Replica: 8617b4db31fe4bc3891e3c9cbbc35324, State: Running, Role: LEADER
I20260928 20:25:18.646126 2644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } }
I20260928 20:25:18.646540 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8617b4db31fe4bc3891e3c9cbbc35324. Latest consensus state: current_term: 1 leader_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } } }
I20260928 20:25:18.646641 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:18.646704 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8617b4db31fe4bc3891e3c9cbbc35324" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34033 } } }
I20260928 20:25:18.646783 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:18.647250 2653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:25:18.647686 2653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:25:18.649252 2653 catalog_manager.cc:1347] Generated new cluster ID: a2e7d32000d6400f88a912cfe4e61326
I20260928 20:25:18.649322 2653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:25:18.675413 2653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:25:18.675652 2653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:25:18.681032 2653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8617b4db31fe4bc3891e3c9cbbc35324: Generated new TSK 0
I20260928 20:25:18.681289 2653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:25:18.684654 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--builtin_ntp_servers=127.24.186.212:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:25:18.771018 2663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.771304 2663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.771350 2663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.772728 2663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.772815 2663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.772863 2663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:25:18.774307 2663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.774591 2663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.774909 2663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.782305 2668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.782322 2669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.782475 2663 server_base.cc:1034] running on GCE node
W20260928 20:25:18.782501 2671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.782861 2663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.783141 2663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.784320 2663 hybrid_clock.cc:648] HybridClock initialized: now 1790627118784297 us; error 45 us; skew 500 ppm
I20260928 20:25:18.785669 2663 webserver.cc:460] Webserver started at http://127.24.186.193:39653/ using document root <none> and password file <none>
I20260928 20:25:18.785928 2663 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.786022 2663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.786183 2663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.787309 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "49f01016941f478385b15550af70323e"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "119dba4b6a12cb2353755bdea9dc0066"
server_key_iv: "b9ba632202dd7f7b40460857170d1bae"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.787691 2663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "49f01016941f478385b15550af70323e"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "119dba4b6a12cb2353755bdea9dc0066"
server_key_iv: "b9ba632202dd7f7b40460857170d1bae"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.789149 2663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:18.790004 2677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.790210 2663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:25:18.790315 2663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "49f01016941f478385b15550af70323e"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "119dba4b6a12cb2353755bdea9dc0066"
server_key_iv: "b9ba632202dd7f7b40460857170d1bae"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.790402 2663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.793824 2663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.794109 2663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.794255 2663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.794463 2663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:18.794781 2663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:18.794840 2663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.794880 2663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:18.794907 2663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.801127 2663 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:33217
I20260928 20:25:18.801211 2790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:33217 every 8 connection(s)
I20260928 20:25:18.801573 2663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:25:18.805051 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.802659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:36729 (local address 127.24.186.254:34033)
0928 20:25:18.802737 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.802740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.802952 (+ 212us) server_negotiation.cc:407] Connection header received
0928 20:25:18.803494 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.803495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.803504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.803547 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.804757 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.804760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.804760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.804764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.804782 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.804786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.804849 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.804950 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.804968 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:25:18.805251 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.802612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34033 (local address 127.24.186.193:36729)
0928 20:25:18.802821 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:18.802855 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:18.803374 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:18.803627 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:18.803631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:18.803641 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:18.804591 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:18.804591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:18.804621 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:18.804650 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:18.804846 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:18.804851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:18.804855 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:25:18.804962 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 20:25:18.805024 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260928 20:25:18.805773 2791 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34033
I20260928 20:25:18.805881 2791 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:18.806054 2791 heartbeater.cc:511] Master 127.24.186.254:34033 requested a full tablet report, sending...
I20260928 20:25:18.806565 2603 ts_manager.cc:194] Registered new tserver with Master: 49f01016941f478385b15550af70323e (127.24.186.193:33217)
I20260928 20:25:18.810249 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2663
I20260928 20:25:18.810377 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:25:18.810570 25323 external_mini_cluster.cc:1428] Setting key 3bb790614038e109795f71f483f62a4c
I20260928 20:25:18.811622 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--builtin_ntp_servers=127.24.186.212:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:25:18.897033 2794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:18.897253 2794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:18.897280 2794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:18.898519 2794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:18.898572 2794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:18.898600 2794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:25:18.900080 2794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:18.900346 2794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:18.900671 2794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:18.905951 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.905973 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:18.905951 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:18.906234 2794 server_base.cc:1034] running on GCE node
I20260928 20:25:18.906570 2794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:18.906797 2794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:18.908035 2794 hybrid_clock.cc:648] HybridClock initialized: now 1790627118907956 us; error 87 us; skew 500 ppm
I20260928 20:25:18.909698 2794 webserver.cc:460] Webserver started at http://127.24.186.194:39253/ using document root <none> and password file <none>
I20260928 20:25:18.910107 2794 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:18.910161 2794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:18.910326 2794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:18.911456 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "09040030811942f38a87bcf00e493f6f"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4f5275caa8f82c912b6c80a93d764e71"
server_key_iv: "44808f53877c30df233e4f3a2ec86e48"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.911900 2794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "09040030811942f38a87bcf00e493f6f"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4f5275caa8f82c912b6c80a93d764e71"
server_key_iv: "44808f53877c30df233e4f3a2ec86e48"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.913326 2794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.914209 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.914434 2794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:18.914539 2794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "09040030811942f38a87bcf00e493f6f"
format_stamp: "Formatted at 2026-09-28 20:25:18 on dist-test-slave-k477"
server_key: "4f5275caa8f82c912b6c80a93d764e71"
server_key_iv: "44808f53877c30df233e4f3a2ec86e48"
server_key_version: "encryptionkey@0"
I20260928 20:25:18.914616 2794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:18.918223 2794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:18.918514 2794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:18.918653 2794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:18.918869 2794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:18.919193 2794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:18.919226 2794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.919247 2794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:18.919298 2794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:18.927168 2794 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:46117
I20260928 20:25:18.927306 2921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:46117 every 8 connection(s)
I20260928 20:25:18.927615 2794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:25:18.930763 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.928410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:58791 (local address 127.24.186.254:34033)
0928 20:25:18.928472 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:18.928474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:18.928746 (+ 272us) server_negotiation.cc:407] Connection header received
0928 20:25:18.929280 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:18.929282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:18.929288 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:18.929345 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:18.930481 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:18.930483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:18.930483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:18.930488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:18.930504 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:18.930509 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:18.930558 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:18.930663 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:25:18.930680 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:25:18.930967 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:18.928402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34033 (local address 127.24.186.194:58791)
0928 20:25:18.928623 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:18.928657 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:18.929158 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:18.929412 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:18.929416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:18.929431 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:18.930322 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:18.930322 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:18.930350 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:18.930377 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:18.930566 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:18.930570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:18.930573 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:25:18.930662 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:25:18.930726 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260928 20:25:18.931408 2922 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34033
I20260928 20:25:18.931533 2922 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:18.931681 2922 heartbeater.cc:511] Master 127.24.186.254:34033 requested a full tablet report, sending...
I20260928 20:25:18.932132 2603 ts_manager.cc:194] Registered new tserver with Master: 09040030811942f38a87bcf00e493f6f (127.24.186.194:46117)
I20260928 20:25:18.937052 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2794
I20260928 20:25:18.937152 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:25:18.937382 25323 external_mini_cluster.cc:1428] Setting key 65785fe082d206bb0146aa83175c645b
I20260928 20:25:18.938441 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--builtin_ntp_servers=127.24.186.212:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 20:25:19.024039 2925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:19.024294 2925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:19.024333 2925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:19.025566 2925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:19.025648 2925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:19.025695 2925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:25:19.027285 2925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:19.027565 2925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:19.027940 2925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:19.033604 2930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:19.033604 2931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:19.033648 2933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:19.033952 2925 server_base.cc:1034] running on GCE node
I20260928 20:25:19.034206 2925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:19.034425 2925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:19.035619 2925 hybrid_clock.cc:648] HybridClock initialized: now 1790627119035614 us; error 41 us; skew 500 ppm
I20260928 20:25:19.037058 2925 webserver.cc:460] Webserver started at http://127.24.186.195:36401/ using document root <none> and password file <none>
I20260928 20:25:19.037317 2925 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:19.037390 2925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:19.037554 2925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:19.038686 2925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "ab759f90f0334fd6a30786ef192729eb"
format_stamp: "Formatted at 2026-09-28 20:25:19 on dist-test-slave-k477"
server_key: "cbbe88257b66cc485144fdb33d2c08bf"
server_key_iv: "2aa77f4f3f165f79b9d35408bf33b675"
server_key_version: "encryptionkey@0"
I20260928 20:25:19.039067 2925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "ab759f90f0334fd6a30786ef192729eb"
format_stamp: "Formatted at 2026-09-28 20:25:19 on dist-test-slave-k477"
server_key: "cbbe88257b66cc485144fdb33d2c08bf"
server_key_iv: "2aa77f4f3f165f79b9d35408bf33b675"
server_key_version: "encryptionkey@0"
I20260928 20:25:19.040611 2925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:25:19.041560 2939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:19.042126 2925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:19.042276 2925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "ab759f90f0334fd6a30786ef192729eb"
format_stamp: "Formatted at 2026-09-28 20:25:19 on dist-test-slave-k477"
server_key: "cbbe88257b66cc485144fdb33d2c08bf"
server_key_iv: "2aa77f4f3f165f79b9d35408bf33b675"
server_key_version: "encryptionkey@0"
I20260928 20:25:19.042376 2925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:19.046161 2925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:19.046478 2925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:19.046640 2925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:19.046854 2925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:19.047223 2925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:19.047286 2925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:19.047326 2925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:19.047353 2925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:19.053056 2925 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38839
I20260928 20:25:19.053161 3052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38839 every 8 connection(s)
I20260928 20:25:19.053479 2925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:25:19.054185 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 2925
I20260928 20:25:19.054280 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:25:19.054493 25323 external_mini_cluster.cc:1428] Setting key e194a20f514ce6627b6ed79917062295
I20260928 20:25:19.056855 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:19.054317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:59247 (local address 127.24.186.254:34033)
0928 20:25:19.054364 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:19.054366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:19.054634 (+ 268us) server_negotiation.cc:407] Connection header received
0928 20:25:19.055398 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:19.055399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:19.055405 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:19.055444 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:19.056625 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:19.056626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:19.056626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:25:19.056631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:19.056645 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:25:19.056648 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:19.056697 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:19.056770 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:25:19.056786 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:25:19.057075 3055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:19.054298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34033 (local address 127.24.186.195:59247)
0928 20:25:19.054497 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:19.054534 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:19.055296 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:19.055519 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:19.055523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:19.055533 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:19.056469 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:25:19.056469 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:19.056497 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:25:19.056524 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:19.056693 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:19.056697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:19.056701 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:25:19.056779 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:25:19.056838 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 20:25:19.057542 3053 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34033
I20260928 20:25:19.057615 3053 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:19.057718 3053 heartbeater.cc:511] Master 127.24.186.254:34033 requested a full tablet report, sending...
I20260928 20:25:19.058040 2603 ts_manager.cc:194] Registered new tserver with Master: ab759f90f0334fd6a30786ef192729eb (127.24.186.195:38839)
I20260928 20:25:19.066638 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 20:25:19.068738 2643 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:25:19.068118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.24.186.254:34033)
0928 20:25:19.068194 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:19.068196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:19.068360 (+ 164us) server_negotiation.cc:407] Connection header received
0928 20:25:19.068405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:19.068406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:19.068412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:19.068464 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:19.068665 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46440: BlockingRecv error: recv got EOF from 127.0.0.1:46440 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:25:19.068838 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2663
I20260928 20:25:19.073653 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2794
I20260928 20:25:19.078661 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2925
I20260928 20:25:19.082991 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 2572
2026-09-28T20:25:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:25:19 dist-test-slave-k477 krb5kdc[3063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:25:19 dist-test-slave-k477 krb5kdc[3063](info): set up 2 sockets
Sep 28 20:25:19 dist-test-slave-k477 krb5kdc[3063](info): commencing operation
krb5kdc: starting...
W20260928 20:25:20.161428 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:25:20.292186 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34035
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:20.378259 3076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:20.378558 3076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:20.378597 3076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:20.379873 3076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:25:20.379956 3076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:20.379979 3076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:20.380008 3076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:25:20.380045 3076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:25:20.381575 3076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:20.381886 3076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:20.382167 3076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:20.387389 3082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:20.387403 3084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.387579 3076 server_base.cc:1034] running on GCE node
W20260928 20:25:20.387379 3081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.387965 3076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:20.388248 3076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:20.389405 3076 hybrid_clock.cc:648] HybridClock initialized: now 1790627120389372 us; error 49 us; skew 500 ppm
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:20.392377 3076 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:25:20.393144 3076 webserver.cc:460] Webserver started at http://127.24.186.254:38085/ using document root <none> and password file <none>
I20260928 20:25:20.393612 3076 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:20.393719 3076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:20.393909 3076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:20.395071 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "820a621d42124f01b6eb4632c928ec53"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "ae8dc9c5656a97888594081915d1de1d"
server_key_iv: "f71df9cfe29bcebd8725b17a5a72aec1"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.395466 3076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "820a621d42124f01b6eb4632c928ec53"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "ae8dc9c5656a97888594081915d1de1d"
server_key_iv: "f71df9cfe29bcebd8725b17a5a72aec1"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.397079 3076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:25:20.398011 3091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.398355 3076 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:20.398451 3076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "820a621d42124f01b6eb4632c928ec53"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "ae8dc9c5656a97888594081915d1de1d"
server_key_iv: "f71df9cfe29bcebd8725b17a5a72aec1"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.398515 3076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:20.416594 3076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:20.420239 3076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:20.420583 3076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:20.425047 3076 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:34035
I20260928 20:25:20.425081 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:34035 every 8 connection(s)
I20260928 20:25:20.425462 3076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:25:20.426534 3144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:20.428076 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 3076
I20260928 20:25:20.428154 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:25:20.428361 25323 external_mini_cluster.cc:1428] Setting key 84a7e3ef4f40bda2afbe22333ffbf437
I20260928 20:25:20.429263 3144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: Bootstrap starting.
I20260928 20:25:20.429913 3144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:20.430239 3144 log.cc:824] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:20.431031 3144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: No bootstrap required, opened a new log
I20260928 20:25:20.433320 3144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } }
I20260928 20:25:20.433518 3144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:20.433559 3144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820a621d42124f01b6eb4632c928ec53, State: Initialized, Role: FOLLOWER
I20260928 20:25:20.433697 3144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } }
I20260928 20:25:20.433975 3144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:25:20.434037 3144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:25:20.434099 3144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:20.435200 3144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } }
I20260928 20:25:20.435334 3144 leader_election.cc:302] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820a621d42124f01b6eb4632c928ec53; no voters:
I20260928 20:25:20.435582 3144 leader_election.cc:288] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:25:20.435604 3149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:25:20.435734 3149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [term 1 LEADER]: Becoming Leader. State: Replica: 820a621d42124f01b6eb4632c928ec53, State: Running, Role: LEADER
I20260928 20:25:20.435854 3149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } }
I20260928 20:25:20.435899 3144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:20.436198 3149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820a621d42124f01b6eb4632c928ec53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } } }
I20260928 20:25:20.436275 3149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:20.436395 3149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820a621d42124f01b6eb4632c928ec53. Latest consensus state: current_term: 1 leader_uuid: "820a621d42124f01b6eb4632c928ec53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820a621d42124f01b6eb4632c928ec53" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 34035 } } }
I20260928 20:25:20.436462 3149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53 [sys.catalog]: This master's current role is: LEADER
I20260928 20:25:20.436919 3154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:25:20.438320 3154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:25:20.440441 3154 catalog_manager.cc:1347] Generated new cluster ID: a682e0b81ff14e2289d58b7ca3cce183
I20260928 20:25:20.440498 3154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:25:20.441428 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:20.428960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.24.186.254:34035)
0928 20:25:20.429141 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:20.429167 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:20.429193 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:25:20.429657 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:20.429664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:20.429713 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:20.429831 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:20.430660 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.431671 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.432819 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.433248 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.433342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:20.436726 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:20.436731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:20.436731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:20.436748 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:20.439419 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.439887 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.439890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.439891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.439984 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.440294 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.440294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.440295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.440618 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:20.440882 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:20.440981 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 20:25:20.441128 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 20:25:20.455636 3154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:25:20.456581 3154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:25:20.470202 3154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: Generated new TSK 0
I20260928 20:25:20.470458 3154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:25:20.512339 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:20.603322 3170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:20.603559 3170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:20.603587 3170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:20.605019 3170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:20.605110 3170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:20.605144 3170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:25:20.606710 3170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:20.606930 3170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:20.607128 3170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:20.613480 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.613569 3170 server_base.cc:1034] running on GCE node
W20260928 20:25:20.613480 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:20.613483 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.613982 3170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:20.614280 3170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:20.615471 3170 hybrid_clock.cc:648] HybridClock initialized: now 1790627120615425 us; error 50 us; skew 500 ppm
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:20.618295 3170 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:25:20.618793 3170 webserver.cc:460] Webserver started at http://127.24.186.193:42229/ using document root <none> and password file <none>
I20260928 20:25:20.619068 3170 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:20.619150 3170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:20.619311 3170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:20.620473 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "5064161e196a4b8c8b42a33b263531f1"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "05cc3185c6dfa3f2160c8acfe7c6e99e"
server_key_iv: "a083bba855f4f02d7401dfd87afb437b"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.620854 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "5064161e196a4b8c8b42a33b263531f1"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "05cc3185c6dfa3f2160c8acfe7c6e99e"
server_key_iv: "a083bba855f4f02d7401dfd87afb437b"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.622293 3170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:25:20.623383 3185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.623728 3170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:20.623970 3170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "5064161e196a4b8c8b42a33b263531f1"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "05cc3185c6dfa3f2160c8acfe7c6e99e"
server_key_iv: "a083bba855f4f02d7401dfd87afb437b"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.624069 3170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:20.667896 3170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:20.670503 3170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:20.670698 3170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:20.670956 3170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:20.671301 3170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:20.671360 3170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.671396 3170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:20.671423 3170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.680339 3170 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:38797
I20260928 20:25:20.680388 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:38797 every 8 connection(s)
I20260928 20:25:20.680742 3170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:20.689373 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 3170
I20260928 20:25:20.689518 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:25:20.689775 25323 external_mini_cluster.cc:1428] Setting key 2fe61bafecf589d83c26a0e5cdecc3b4
I20260928 20:25:20.692611 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:20.681540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:39751 (local address 127.24.186.254:34035)
0928 20:25:20.681616 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:20.681618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:20.681913 (+ 295us) server_negotiation.cc:407] Connection header received
0928 20:25:20.682470 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:20.682471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:20.682483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:20.682537 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:20.683516 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.684074 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.685513 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.685839 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.685943 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:20.688658 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:20.688660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:20.688661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:20.688670 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:20.690617 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.691212 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.691213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.691214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.691285 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.691645 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.691646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.691647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.691784 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:20.691933 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:20.692363 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 20:25:20.692484 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:25:20.692628 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:20.681488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34035 (local address 127.24.186.193:39751)
0928 20:25:20.681706 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:20.681767 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:20.682339 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:20.682602 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:20.682605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:20.683210 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:20.683412 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:20.683416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.684178 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.684179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:20.685387 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:20.685390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.685976 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.685976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:20.686093 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:20.686625 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:20.686634 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:20.688529 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:20.690722 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:20.690734 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:20.690738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:20.691109 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:20.691365 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:20.691366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:20.691367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:20.691532 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:20.691940 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:20.691947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:20.692149 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 20:25:20.692288 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 20:25:20.692445 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260928 20:25:20.693410 3299 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34035
I20260928 20:25:20.693508 3299 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:20.693724 3299 heartbeater.cc:511] Master 127.24.186.254:34035 requested a full tablet report, sending...
I20260928 20:25:20.694404 3108 ts_manager.cc:194] Registered new tserver with Master: 5064161e196a4b8c8b42a33b263531f1 (127.24.186.193:38797)
I20260928 20:25:20.695353 3108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:39751
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:25:20.724011 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:20.810438 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:20.810648 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:20.810673 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:20.811898 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:20.811944 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:20.811971 3304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:25:20.813548 3304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:20.813807 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:20.814073 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:20.819438 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:20.819449 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.819686 3304 server_base.cc:1034] running on GCE node
W20260928 20:25:20.819466 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.820058 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:20.820314 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:20.821470 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790627120821453 us; error 31 us; skew 500 ppm
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:20.824148 3304 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:25:20.824643 3304 webserver.cc:460] Webserver started at http://127.24.186.194:46857/ using document root <none> and password file <none>
I20260928 20:25:20.824857 3304 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:20.824899 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:20.824991 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:20.826035 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "67df6feef3774b98b07eb80ffbb4d3c6"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "5bebbb4d15c0151e19e6a0f48c183bd3"
server_key_iv: "3c20f6eab1fa1fa9edc9e102e1274f7e"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.826349 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "67df6feef3774b98b07eb80ffbb4d3c6"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "5bebbb4d15c0151e19e6a0f48c183bd3"
server_key_iv: "3c20f6eab1fa1fa9edc9e102e1274f7e"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.827721 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:25:20.828619 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.828860 3304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:20.828966 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "67df6feef3774b98b07eb80ffbb4d3c6"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "5bebbb4d15c0151e19e6a0f48c183bd3"
server_key_iv: "3c20f6eab1fa1fa9edc9e102e1274f7e"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.829056 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:20.842789 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:20.845098 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:20.845281 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:20.845551 3304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:20.845884 3304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:20.845943 3304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.845984 3304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:20.846006 3304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.853217 3304 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:43091
I20260928 20:25:20.853304 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:43091 every 8 connection(s)
I20260928 20:25:20.853650 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:25:20.859941 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 3304
I20260928 20:25:20.860069 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:25:20.860286 25323 external_mini_cluster.cc:1428] Setting key 71c191673fea3f3433cc8adea63211f9
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:20.864714 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:20.854769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:43705 (local address 127.24.186.254:34035)
0928 20:25:20.854870 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:20.854872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:20.855081 (+ 209us) server_negotiation.cc:407] Connection header received
0928 20:25:20.855568 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:20.855569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:20.855580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:20.855643 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:20.856586 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.857095 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.858161 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.858348 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.858411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:20.860905 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:20.860907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:20.860907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:20.860915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:20.862907 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.863544 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.863545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.863545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.863606 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:20.863993 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:20.863994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:20.863995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:20.864141 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:20.864231 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:20.864528 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 20:25:20.864618 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 20:25:20.864926 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:20.854752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34035 (local address 127.24.186.194:43705)
0928 20:25:20.854923 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:20.854966 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:20.855479 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:20.855697 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:20.855700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:20.856302 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:20.856500 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:20.856504 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.857180 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.857181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:20.857976 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:20.857978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:20.858432 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:20.858433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:20.858532 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:20.859123 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:20.859133 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:20.860835 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:20.863027 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:20.863037 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:20.863040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:20.863416 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:20.863693 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:20.863694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:20.863695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:20.863899 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:20.864220 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:20.864226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:20.864375 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 20:25:20.864493 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 20:25:20.864676 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260928 20:25:20.865440 3433 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34035
I20260928 20:25:20.865543 3433 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:20.865742 3433 heartbeater.cc:511] Master 127.24.186.254:34035 requested a full tablet report, sending...
I20260928 20:25:20.866164 3108 ts_manager.cc:194] Registered new tserver with Master: 67df6feef3774b98b07eb80ffbb4d3c6 (127.24.186.194:43091)
I20260928 20:25:20.866570 3108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:43705
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:25:20.891616 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:25:20.976864 3438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:25:20.977108 3438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:25:20.977133 3438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:25:20.978271 3438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:25:20.978307 3438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:25:20.978333 3438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:25:20.979729 3438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:25:20.980042 3438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:25:20.980247 3438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:25:20.985684 3446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:25:20.985670 3444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.985703 3438 server_base.cc:1034] running on GCE node
W20260928 20:25:20.985673 3443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:25:20.986209 3438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:25:20.986512 3438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:25:20.987676 3438 hybrid_clock.cc:648] HybridClock initialized: now 1790627120987652 us; error 66 us; skew 500 ppm
Sep 28 20:25:20 dist-test-slave-k477 krb5kdc[3063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:25:20.990219 3438 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:25:20.990700 3438 webserver.cc:460] Webserver started at http://127.24.186.195:36335/ using document root <none> and password file <none>
I20260928 20:25:20.990974 3438 fs_manager.cc:359] Metadata directory not provided
I20260928 20:25:20.991031 3438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:25:20.991163 3438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:25:20.992326 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "17d3f2e714604edeb0924601cc296c44"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "3db87fe22ce794ed6b1826fee4e42bb6"
server_key_iv: "76c9ff2e13e14f4efcbf1269c2da696a"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.992693 3438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "17d3f2e714604edeb0924601cc296c44"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "3db87fe22ce794ed6b1826fee4e42bb6"
server_key_iv: "76c9ff2e13e14f4efcbf1269c2da696a"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.994822 3438 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:25:20.995754 3453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:20.996030 3438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:25:20.996165 3438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "17d3f2e714604edeb0924601cc296c44"
format_stamp: "Formatted at 2026-09-28 20:25:20 on dist-test-slave-k477"
server_key: "3db87fe22ce794ed6b1826fee4e42bb6"
server_key_iv: "76c9ff2e13e14f4efcbf1269c2da696a"
server_key_version: "encryptionkey@0"
I20260928 20:25:20.996259 3438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:25:21.020116 3438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:25:21.022388 3438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:25:21.022576 3438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:25:21.022930 3438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:25:21.023305 3438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:25:21.023370 3438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:21.023406 3438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:25:21.023433 3438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:25:21.030856 3438 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38851
I20260928 20:25:21.030891 3566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38851 every 8 connection(s)
I20260928 20:25:21.031344 3438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:25:21.037381 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 3438
I20260928 20:25:21.037511 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:25:21.037698 25323 external_mini_cluster.cc:1428] Setting key 179255c806cdbec741320cd4cece019c
Sep 28 20:25:21 dist-test-slave-k477 krb5kdc[3063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627120, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:25:21.042076 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.032184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:35639 (local address 127.24.186.254:34035)
0928 20:25:21.032291 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.032294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.032580 (+ 286us) server_negotiation.cc:407] Connection header received
0928 20:25:21.033068 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.033069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.033079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.033148 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:21.034382 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.034895 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.036161 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.036360 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.036478 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.039021 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:21.039023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:21.039023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:21.039030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:21.040579 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:21.041020 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:21.041021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:21.041022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:21.041080 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:21.041362 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:21.041363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:21.041363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:21.041478 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:21.041532 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.041903 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.041982 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:25:21.042238 3569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.032170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:34035 (local address 127.24.186.195:35639)
0928 20:25:21.032366 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.032408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.032958 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.033261 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.033265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.034057 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:25:21.034260 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.034265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.034993 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.034993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.036064 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.036066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.036474 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.036475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.036601 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.037203 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:25:21.037212 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:25:21.038934 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:25:21.040681 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:21.040687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:21.040689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:21.040941 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:21.041182 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:25:21.041183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:25:21.041184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:25:21.041295 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:25:21.041567 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:25:21.041574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:25:21.041720 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.041835 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.041971 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":126,"threads_started":1}
I20260928 20:25:21.042753 3567 heartbeater.cc:347] Connected to a master server at 127.24.186.254:34035
I20260928 20:25:21.042865 3567 heartbeater.cc:464] Registering TS with master...
I20260928 20:25:21.043025 3567 heartbeater.cc:511] Master 127.24.186.254:34035 requested a full tablet report, sending...
I20260928 20:25:21.043411 3108 ts_manager.cc:194] Registered new tserver with Master: 17d3f2e714604edeb0924601cc296c44 (127.24.186.195:38851)
I20260928 20:25:21.043761 3108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:35639
I20260928 20:25:21.050446 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:25:21.058280 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.051842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.24.186.254:34035)
0928 20:25:21.051913 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.051915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.052017 (+ 102us) server_negotiation.cc:407] Connection header received
0928 20:25:21.052147 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.052148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.052159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.052212 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:25:21.052805 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.053313 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.054075 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.054290 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.054387 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.055616 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:25:21.055618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:25:21.055618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:25:21.055625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:25:21.056996 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:21.057393 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:21.057394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:21.057395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:21.057454 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:25:21.057674 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:25:21.057675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:25:21.057675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:25:21.057828 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:25:21.057937 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.058067 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.058155 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:25:21.060442 3108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:25:21.062165 3108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:25:21.068951 3578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.064752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:43091 (local address 127.0.0.1:56648)
0928 20:25:21.064917 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.064957 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.065022 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.065372 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.065375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.065377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:25:21.065436 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.065437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.066612 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.066613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.067737 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.067739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.068620 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.068621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.068703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.068717 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.068814 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.068870 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260928 20:25:21.068964 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.065007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.24.186.194:43091)
0928 20:25:21.065185 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.065190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.065202 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:25:21.065242 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.065245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.065260 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.065343 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:25:21.065623 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.066493 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.067857 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.068526 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.068624 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.068684 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.068829 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.068869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260928 20:25:21.069594 3580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.064754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38797 (local address 127.0.0.1:60154)
0928 20:25:21.065566 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.065575 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.065629 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.065907 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.065908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.065910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:25:21.065950 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.065952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.067183 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.067184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.068221 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.068224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.069322 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.069323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.069400 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.069405 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.069461 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.069510 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260928 20:25:21.070044 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.064875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.24.186.193:38797)
0928 20:25:21.065195 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.065198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.065687 (+ 489us) server_negotiation.cc:407] Connection header received
0928 20:25:21.065745 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.065747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.065761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.065830 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:25:21.066040 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.067098 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.068308 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.069137 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.069231 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.069285 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.069491 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.069921 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":82,"threads_started":1}
I20260928 20:25:21.069487 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.064752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38851 (local address 127.0.0.1:58590)
0928 20:25:21.065419 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.065428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.065469 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.065746 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.065747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.065748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:25:21.065795 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.065797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.067148 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.067149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.068128 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.068130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.069238 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.069239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.069299 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.069303 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.069359 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.069403 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260928 20:25:21.070763 3367 tablet_service.cc:1467] Processing CreateTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 (DEFAULT_TABLE table=test-table [id=6aaa120fcdcc404da4311232258ee0e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:25:21.071033 3233 tablet_service.cc:1467] Processing CreateTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 (DEFAULT_TABLE table=test-table [id=6aaa120fcdcc404da4311232258ee0e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:25:21.071331 3367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b714fa3d0d4adf923ae7d6f131b2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:21.071367 3233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b714fa3d0d4adf923ae7d6f131b2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:21.072016 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.064992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.24.186.195:38851)
0928 20:25:21.065159 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.065163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.065502 (+ 339us) server_negotiation.cc:407] Connection header received
0928 20:25:21.065547 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.065549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.065565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.065668 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:25:21.065874 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.067016 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.068223 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.069138 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.069229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.069282 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.069392 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.071935 (+ 2543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 20:25:21.072769 3501 tablet_service.cc:1467] Processing CreateTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 (DEFAULT_TABLE table=test-table [id=6aaa120fcdcc404da4311232258ee0e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:25:21.073328 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88b714fa3d0d4adf923ae7d6f131b2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:25:21.076277 3584 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Bootstrap starting.
I20260928 20:25:21.077051 3586 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Bootstrap starting.
I20260928 20:25:21.077283 3584 tablet_bootstrap.cc:654] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:21.077339 3585 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Bootstrap starting.
I20260928 20:25:21.077896 3586 tablet_bootstrap.cc:654] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:21.077998 3584 log.cc:824] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:21.078557 3585 tablet_bootstrap.cc:654] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Neither blocks nor log segments found. Creating new log.
I20260928 20:25:21.078864 3586 log.cc:824] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:21.078912 3584 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: No bootstrap required, opened a new log
I20260928 20:25:21.078994 3584 ts_tablet_manager.cc:1397] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:25:21.079262 3585 log.cc:824] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Log is configured to *not* fsync() on all Append() calls
I20260928 20:25:21.079757 3586 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: No bootstrap required, opened a new log
I20260928 20:25:21.079869 3586 ts_tablet_manager.cc:1397] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:25:21.080080 3585 tablet_bootstrap.cc:492] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: No bootstrap required, opened a new log
I20260928 20:25:21.080168 3585 ts_tablet_manager.cc:1397] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:25:21.081211 3584 raft_consensus.cc:348] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.081403 3584 raft_consensus.cc:374] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:21.081437 3584 raft_consensus.cc:729] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5064161e196a4b8c8b42a33b263531f1, State: Initialized, Role: FOLLOWER
I20260928 20:25:21.081593 3584 consensus_queue.cc:260] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.081890 3584 ts_tablet_manager.cc:1428] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:25:21.081920 3586 raft_consensus.cc:348] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.082048 3299 heartbeater.cc:503] Master 127.24.186.254:34035 was elected leader, sending a full tablet report...
I20260928 20:25:21.082122 3586 raft_consensus.cc:374] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:21.082068 3585 raft_consensus.cc:348] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.082170 3586 raft_consensus.cc:729] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67df6feef3774b98b07eb80ffbb4d3c6, State: Initialized, Role: FOLLOWER
I20260928 20:25:21.082216 3585 raft_consensus.cc:374] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:25:21.082255 3585 raft_consensus.cc:729] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d3f2e714604edeb0924601cc296c44, State: Initialized, Role: FOLLOWER
I20260928 20:25:21.082291 3586 consensus_queue.cc:260] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.082420 3585 consensus_queue.cc:260] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.082571 3586 ts_tablet_manager.cc:1428] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:25:21.082643 3433 heartbeater.cc:503] Master 127.24.186.254:34035 was elected leader, sending a full tablet report...
I20260928 20:25:21.082682 3585 ts_tablet_manager.cc:1428] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 20:25:21.082785 3567 heartbeater.cc:503] Master 127.24.186.254:34035 was elected leader, sending a full tablet report...
W20260928 20:25:21.104270 3434 tablet.cc:2369] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:25:21.181663 3300 tablet.cc:2369] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:25:21.282045 3568 tablet.cc:2369] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:25:21.336463 3591 raft_consensus.cc:482] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:25:21.336658 3591 raft_consensus.cc:504] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.337064 3591 leader_election.cc:288] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17d3f2e714604edeb0924601cc296c44 (127.24.186.195:38851), 5064161e196a4b8c8b42a33b263531f1 (127.24.186.193:38797)
I20260928 20:25:21.340456 3581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.337743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:38313 (local address 127.24.186.193:38797)
0928 20:25:21.337802 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.337804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.337813 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:25:21.337842 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.337843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.337854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.337940 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:25:21.338089 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.338550 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.339613 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.340075 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.340167 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.340177 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.340337 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.340377 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:25:21.340456 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.337324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38797 (local address 127.24.186.194:38313)
0928 20:25:21.337432 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.337452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.337765 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.337942 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.337942 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.337944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:25:21.338011 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.338012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.338649 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.338650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.339499 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.339501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.340166 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.340168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.340248 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.340252 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.340331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.340375 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":31}
I20260928 20:25:21.340561 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.337445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:39717 (local address 127.24.186.195:38851)
0928 20:25:21.337519 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.337521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.337563 (+ 42us) server_negotiation.cc:407] Connection header received
0928 20:25:21.337771 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.337772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.337783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.337835 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:25:21.338010 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.338506 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.339731 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.340147 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.340245 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.340254 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.340418 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.340458 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:25:21.340619 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.337323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38851 (local address 127.24.186.194:39717)
0928 20:25:21.337503 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.337514 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:25:21.337598 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:25:21.337844 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:25:21.337845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:25:21.337847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:25:21.337929 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.337931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.338610 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.338611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.339592 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:25:21.339595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.340253 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.340254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:25:21.340329 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.340332 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:25:21.340410 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:25:21.340450 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":235,"thread_start_us":69,"threads_started":1}
I20260928 20:25:21.340852 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b714fa3d0d4adf923ae7d6f131b2c6" candidate_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064161e196a4b8c8b42a33b263531f1" is_pre_election: true
I20260928 20:25:21.341048 3253 raft_consensus.cc:2454] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67df6feef3774b98b07eb80ffbb4d3c6 in term 0.
I20260928 20:25:21.341151 3521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b714fa3d0d4adf923ae7d6f131b2c6" candidate_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d3f2e714604edeb0924601cc296c44" is_pre_election: true
I20260928 20:25:21.341310 3521 raft_consensus.cc:2454] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67df6feef3774b98b07eb80ffbb4d3c6 in term 0.
I20260928 20:25:21.341326 3323 leader_election.cc:302] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5064161e196a4b8c8b42a33b263531f1, 67df6feef3774b98b07eb80ffbb4d3c6; no voters:
I20260928 20:25:21.341447 3591 raft_consensus.cc:2781] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:25:21.341487 3591 raft_consensus.cc:482] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:25:21.341511 3591 raft_consensus.cc:3037] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:21.342290 3591 raft_consensus.cc:504] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.342526 3591 leader_election.cc:288] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [CANDIDATE]: Term 1 election: Requested vote from peers 17d3f2e714604edeb0924601cc296c44 (127.24.186.195:38851), 5064161e196a4b8c8b42a33b263531f1 (127.24.186.193:38797)
I20260928 20:25:21.342623 3521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b714fa3d0d4adf923ae7d6f131b2c6" candidate_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17d3f2e714604edeb0924601cc296c44"
I20260928 20:25:21.342697 3521 raft_consensus.cc:3037] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:21.342718 3253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88b714fa3d0d4adf923ae7d6f131b2c6" candidate_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5064161e196a4b8c8b42a33b263531f1"
I20260928 20:25:21.342824 3253 raft_consensus.cc:3037] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:25:21.343446 3521 raft_consensus.cc:2454] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67df6feef3774b98b07eb80ffbb4d3c6 in term 1.
I20260928 20:25:21.343628 3253 raft_consensus.cc:2454] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67df6feef3774b98b07eb80ffbb4d3c6 in term 1.
I20260928 20:25:21.343629 3320 leader_election.cc:302] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17d3f2e714604edeb0924601cc296c44, 67df6feef3774b98b07eb80ffbb4d3c6; no voters:
I20260928 20:25:21.343791 3591 raft_consensus.cc:2781] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:25:21.343966 3591 raft_consensus.cc:686] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 67df6feef3774b98b07eb80ffbb4d3c6, State: Running, Role: LEADER
I20260928 20:25:21.344074 3591 consensus_queue.cc:237] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } }
I20260928 20:25:21.344920 3107 catalog_manager.cc:5495] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67df6feef3774b98b07eb80ffbb4d3c6 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67df6feef3774b98b07eb80ffbb4d3c6" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 43091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 } health_report { overall_health: UNKNOWN } } }
I20260928 20:25:21.354935 3583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.351632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.24.186.194:43091)
0928 20:25:21.351673 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:25:21.351675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:25:21.351683 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:25:21.351723 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:25:21.351724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:25:21.351749 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:25:21.351898 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:25:21.352180 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.352670 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.353880 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:25:21.354142 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:25:21.354225 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:25:21.354405 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:25:21.354515 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:25:21.354580 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:25:21.354678 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 20:25:21.354815 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:25:21.357121 3521 raft_consensus.cc:1261] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 1 FOLLOWER]: Refusing update from remote peer 67df6feef3774b98b07eb80ffbb4d3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:25:21.357223 3253 raft_consensus.cc:1261] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 5064161e196a4b8c8b42a33b263531f1 [term 1 FOLLOWER]: Refusing update from remote peer 67df6feef3774b98b07eb80ffbb4d3c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:25:21.357460 3594 consensus_queue.cc:1035] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17d3f2e714604edeb0924601cc296c44" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:25:21.357620 3591 consensus_queue.cc:1035] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5064161e196a4b8c8b42a33b263531f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:25:21.360082 3597 mvcc.cc:205] Tried to move back new op lower bound from 7334408689075974144 to 7334408689026101248. Current Snapshot: MvccSnapshot[applied={T|T < 7334408689075974144}]
I20260928 20:25:21.360288 3599 mvcc.cc:205] Tried to move back new op lower bound from 7334408689075974144 to 7334408689026101248. Current Snapshot: MvccSnapshot[applied={T|T < 7334408689075974144}]
I20260928 20:25:21.360859 3598 mvcc.cc:205] Tried to move back new op lower bound from 7334408689075974144 to 7334408689026101248. Current Snapshot: MvccSnapshot[applied={T|T < 7334408689075974144}]
I20260928 20:25:21.362092 3107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:25:21.362202 3107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:25:21.362978 3107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 820a621d42124f01b6eb4632c928ec53: Sending DeleteTablet for 3 replicas of tablet 88b714fa3d0d4adf923ae7d6f131b2c6
I20260928 20:25:21.363314 3501 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:25:21 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:58590
I20260928 20:25:21.363344 3233 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:25:21 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:60154
I20260928 20:25:21.363525 3367 tablet_service.cc:1514] Processing DeleteTablet for tablet 88b714fa3d0d4adf923ae7d6f131b2c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:25:21 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56648
I20260928 20:25:21.363745 3609 tablet_replica.cc:330] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: stopping tablet replica
I20260928 20:25:21.363842 3607 tablet_replica.cc:330] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: stopping tablet replica
I20260928 20:25:21.363926 3609 raft_consensus.cc:2229] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:25:21.364022 3607 raft_consensus.cc:2229] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:25:21.364097 3609 raft_consensus.cc:2258] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:25:21.364161 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 3170
I20260928 20:25:21.364729 3607 raft_consensus.cc:2258] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:25:21.365209 3607 ts_tablet_manager.cc:1905] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:25:21.366107 3609 ts_tablet_manager.cc:1905] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:25:21.367235 3607 ts_tablet_manager.cc:1918] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:25:21.367375 3607 log.cc:1196] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/88b714fa3d0d4adf923ae7d6f131b2c6
I20260928 20:25:21.367871 3609 ts_tablet_manager.cc:1918] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:25:21.368028 3609 log.cc:1196] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/88b714fa3d0d4adf923ae7d6f131b2c6
I20260928 20:25:21.368045 3607 ts_tablet_manager.cc:1939] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 17d3f2e714604edeb0924601cc296c44: Deleting consensus metadata
I20260928 20:25:21.368381 3609 ts_tablet_manager.cc:1939] T 88b714fa3d0d4adf923ae7d6f131b2c6 P 67df6feef3774b98b07eb80ffbb4d3c6: Deleting consensus metadata
I20260928 20:25:21.368885 3093 catalog_manager.cc:4841] TS 67df6feef3774b98b07eb80ffbb4d3c6 (127.24.186.194:43091): tablet 88b714fa3d0d4adf923ae7d6f131b2c6 (table test-table [id=6aaa120fcdcc404da4311232258ee0e4]) successfully deleted
I20260928 20:25:21.369289 3092 catalog_manager.cc:4841] TS 17d3f2e714604edeb0924601cc296c44 (127.24.186.195:38851): tablet 88b714fa3d0d4adf923ae7d6f131b2c6 (table test-table [id=6aaa120fcdcc404da4311232258ee0e4]) successfully deleted
W20260928 20:25:21.370743 3095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:38797 (error 108)
I20260928 20:25:21.371361 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:25:21.371214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:38797 (local address 127.0.0.1:60162)
0928 20:25:21.371281 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:25:21.371326 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:38797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 20:25:21.371420 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 3304
W20260928 20:25:21.372042 3095 catalog_manager.cc:4584] TS 5064161e196a4b8c8b42a33b263531f1 (127.24.186.193:38797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88b714fa3d0d4adf923ae7d6f131b2c6: Network error: Client connection negotiation failed: client connection to 127.24.186.193:38797: connect: Connection refused (error 111)
I20260928 20:25:21.376606 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 3438
I20260928 20:25:21.381433 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 3076
2026-09-28T20:25:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2300 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:25:21 dist-test-slave-k477 krb5kdc[3613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:25:21 dist-test-slave-k477 krb5kdc[3613](info): set up 2 sockets
Sep 28 20:25:21 dist-test-slave-k477 krb5kdc[3613](info): commencing operation
krb5kdc: starting...
W20260928 20:25:22.453542 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:25:22 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:25:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangeradmin_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangeradmin_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangerlookup_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangerlookup_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
I20260928 20:25:22.639271 25323 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9jCLVo/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/postgres -l logfile start
2026-09-28 20:25:25.574 UTC [3643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 20:25:25.574 UTC [3643] LOG: listening on IPv4 address "127.24.186.212", port 50673
2026-09-28 20:25:25.579 UTC [3643] LOG: listening on Unix socket "/tmp/.s.PGSQL.50673"
2026-09-28 20:25:25.618 UTC [3646] LOG: database system was shut down at 2026-09-28 20:25:23 UTC
2026-09-28 20:25:25.624 UTC [3643] LOG: database system is ready to accept connections
I20260928 20:25:26.601733 25323 mini_postgres.cc:96] Postgres bound to 50673
2026-09-28 20:25:26.609 UTC [3654] FATAL: database "slave" does not exist
127.24.186.212:50673 - accepting connections
I20260928 20:25:26.610052 25323 mini_ranger.cc:161] Starting Ranger...
I20260928 20:25:26.627650 25323 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 20:25:27.226279 25323 mini_ranger.cc:87] Created ranger Postgres database
I20260928 20:25:27.226378 25323 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin
2026-09-28 20:25:27,561 [I] DB FLAVOR :POSTGRES
2026-09-28 20:25:27,562 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:25:27,562 [I] Checking connection..
2026-09-28 20:25:27,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 20:25:27,845 [I] Checking connection passed.
2026-09-28 20:25:27,845 [I] --------- Verifying version history table ---------
2026-09-28 20:25:27,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:25:28,096 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:25:28,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:25:28,349 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:25:28,349 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 20:25:28,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 20:25:28.632 UTC [3718] WARNING: there is no transaction in progress
2026-09-28 20:25:28,639 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 20:25:28,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:25:28,923 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 20:25:28,924 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:25:28,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 20:25:29,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 20:25:29,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k477.c.gcp-upstream.internal','N') ;"
2026-09-28 20:25:29,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:25:30,018 [I] Table x_portal_user does not exist in database ranger
2026-09-28 20:25:30,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:25:30,283 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 20:25:30,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:25:30,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:25:30,844 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 20:25:30,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 20:25:31.926 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:31.946 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:31.966 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:31.985 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.005 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.024 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.090 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.099 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.116 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.128 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.142 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.160 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.171 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.179 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.188 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32.197 UTC [3826] WARNING: there is no transaction in progress
2026-09-28 20:25:32,970 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 20:25:32,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 20:25:33,211 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 20:25:33,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:25:33,463 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 20:25:33,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:25:33,725 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 20:25:33,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:25:34,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:25:34,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k477.c.gcp-upstream.internal';"
2026-09-28 20:25:34,525 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 20:25:34,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 20:25:34,790 [I] DB_PATCHES have already been applied
I20260928 20:25:34.796657 25323 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9jCLVo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 20:25:34.796736 25323 mini_ranger.cc:193] Using host: 127.24.186.212
I20260928 20:25:34.800040 25323 mini_ranger.cc:227] Ranger admin URL: http://127.24.186.212:41103
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:25:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 20:25:36.039368 25323 mini_ranger.cc:160] Time spent starting Ranger: real 9.429s user 0.003s sys 0.006s
Sep 28, 2026 8:25:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:25:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:25:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:25:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:25:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 20:26:00.075203 25323 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangerkms_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/rangerkms_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.186.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.186.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 20:26:00 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627160, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 20:26:00.166143 25323 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 20:26:00.179082 25323 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 20:26:01.194978 25323 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 20:26:01.195060 25323 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms
I20260928 20:26:01.195073 25323 mini_ranger_kms.cc:227] Using postgres at 127.24.186.212:50673
2026-09-28 20:26:01,437 [I] DB FLAVOR :POSTGRES
2026-09-28 20:26:01,438 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:26:01,438 [I] Checking connection
2026-09-28 20:26:01,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 20:26:01,734 [I] connection success
2026-09-28 20:26:01,735 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 20:26:01,735 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 20:26:01,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 20:26:01,996 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 20:26:01,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:26:01,996 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 20:26:01,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:50673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 20:26:02,280 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 20:26:02.843578 25323 mini_ranger_kms.cc:310] Created kms service
I20260928 20:26:03.077157 25323 mini_ranger_kms.cc:326] Created kudu user
I20260928 20:26:03.121140 25323 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 20:26:03 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627160, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:03.578410 25323 mini_ranger_kms.cc:384] Added ranger policy
I20260928 20:26:03.578611 25323 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms:/tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 20:26:03.578637 25323 mini_ranger_kms.cc:240] Using host: 127.24.186.212
I20260928 20:26:03.583313 25323 mini_ranger_kms.cc:276] Ranger KMS PID: 4007
I20260928 20:26:03.583557 25323 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.186.212:50289
Sep 28, 2026 8:26:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:26:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:26:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:26:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:26:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:26:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 20:26:05 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790627165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:05.917403 25323 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.751s user 0.004s sys 0.216s
I20260928 20:26:05.917523 25323 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 20:26:05.917623 25323 mini_ranger_kms.cc:401] 127.24.186.212:50289/kms/v1/keys
Sep 28 20:26:06 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
Sep 28 20:26:06 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627165, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
Sep 28 20:26:09 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:26:09.300202 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46757
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:46757
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:50289/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:26:09.415060 4057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:26:09.415285 4057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:26:09.415309 4057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:26:09.416659 4057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:26:09.416710 4057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:26:09.416724 4057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:26:09.416738 4057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:26:09.416749 4057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:26:09.418419 4057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.186.212:50289/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:46757
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:26:09.418711 4057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:26:09.418957 4057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:26:09.424834 4063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:09.424834 4062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:09.425025 4057 server_base.cc:1034] running on GCE node
W20260928 20:26:09.425051 4065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:09.425634 4057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:26:09.426038 4057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:26:09.427206 4057 hybrid_clock.cc:648] HybridClock initialized: now 1790627169427187 us; error 37 us; skew 500 ppm
Sep 28 20:26:09 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:09.430033 4057 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:26:09.430554 4057 webserver.cc:460] Webserver started at http://127.24.186.254:40821/ using document root <none> and password file <none>
I20260928 20:26:09.430802 4057 fs_manager.cc:359] Metadata directory not provided
I20260928 20:26:09.430858 4057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:26:09.431000 4057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:26:09 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627169, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:09.488029 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "e2ab2724669d4c7aa34318b5b60aa905"
format_stamp: "Formatted at 2026-09-28 20:26:09 on dist-test-slave-k477"
server_key: "130a737d93bff4b63fcc6439298b230c"
server_key_iv: "cd7a42fc5b60c82748b3135227dac379"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:09.488462 4057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "e2ab2724669d4c7aa34318b5b60aa905"
format_stamp: "Formatted at 2026-09-28 20:26:09 on dist-test-slave-k477"
server_key: "130a737d93bff4b63fcc6439298b230c"
server_key_iv: "cd7a42fc5b60c82748b3135227dac379"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:09.489913 4057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:26:09.505525 4072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:09.505712 4057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:26:09.505810 4057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "e2ab2724669d4c7aa34318b5b60aa905"
format_stamp: "Formatted at 2026-09-28 20:26:09 on dist-test-slave-k477"
server_key: "130a737d93bff4b63fcc6439298b230c"
server_key_iv: "cd7a42fc5b60c82748b3135227dac379"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:09.505939 4057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:26:09.532342 4057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:26:09.535503 4057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:26:09.535699 4057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:26:09.541196 4057 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:46757
I20260928 20:26:09.541286 4124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:46757 every 8 connection(s)
I20260928 20:26:09.541649 4057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:26:09.542331 4125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:26:09.544775 4125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: Bootstrap starting.
I20260928 20:26:09.545317 4125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: Neither blocks nor log segments found. Creating new log.
I20260928 20:26:09.545552 4125 log.cc:824] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: Log is configured to *not* fsync() on all Append() calls
I20260928 20:26:09.546239 4125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: No bootstrap required, opened a new log
I20260928 20:26:09.548033 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 4057
I20260928 20:26:09.548161 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:26:09.548352 25323 external_mini_cluster.cc:1428] Setting key 39205957b995de9c15e64e1303a10926
I20260928 20:26:09.548580 4125 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } }
I20260928 20:26:09.548801 4125 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:26:09.549088 4125 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ab2724669d4c7aa34318b5b60aa905, State: Initialized, Role: FOLLOWER
I20260928 20:26:09.549252 4125 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } }
I20260928 20:26:09.549340 4125 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:26:09.549372 4125 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:26:09.549415 4125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:26:09.550073 4125 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } }
I20260928 20:26:09.550195 4125 leader_election.cc:302] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2ab2724669d4c7aa34318b5b60aa905; no voters:
I20260928 20:26:09.550372 4125 leader_election.cc:288] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:26:09.550437 4130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:26:09.550589 4130 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [term 1 LEADER]: Becoming Leader. State: Replica: e2ab2724669d4c7aa34318b5b60aa905, State: Running, Role: LEADER
I20260928 20:26:09.550623 4125 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:26:09.550750 4130 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } }
I20260928 20:26:09.551054 4132 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2ab2724669d4c7aa34318b5b60aa905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } } }
I20260928 20:26:09.551159 4132 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [sys.catalog]: This master's current role is: LEADER
I20260928 20:26:09.551052 4131 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2ab2724669d4c7aa34318b5b60aa905. Latest consensus state: current_term: 1 leader_uuid: "e2ab2724669d4c7aa34318b5b60aa905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2ab2724669d4c7aa34318b5b60aa905" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46757 } } }
I20260928 20:26:09.551329 4131 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905 [sys.catalog]: This master's current role is: LEADER
I20260928 20:26:09.551457 4135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:26:09.552448 4135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:26:09.552762 4125 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 28 20:26:09 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:26:09.559908 4135 catalog_manager.cc:1347] Generated new cluster ID: 27abe95297ab4cdcbbb9c613e29fad84
I20260928 20:26:09.560060 4135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:26:09.564755 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:09.548932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.24.186.254:46757)
0928 20:26:09.549114 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:09.549123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:09.549157 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:26:09.549278 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:09.549285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:09.549346 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:09.549501 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:09.550166 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:09.552414 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:09.553494 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:09.556467 (+ 2973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:09.556813 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:09.560014 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:09.560019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:09.560019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:09.560031 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:09.562337 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:09.562813 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:09.562817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:09.562818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:09.562936 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:09.563140 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:09.563141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:09.563141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:09.563531 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:09.563735 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:09.564344 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 20:26:09.564487 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 20:26:09.572882 4135 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:26:09.573693 4135 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:26:09.584302 4135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: Generated new TSK 0
I20260928 20:26:09.584547 4135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:26:10.291597 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:50289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:26:10.423439 4170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:26:10.423758 4170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:26:10.423839 4170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:26:10.425762 4170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:26:10.425839 4170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:26:10.425885 4170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:26:10.428263 4170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.186.212:50289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:26:10.428550 4170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:26:10.428793 4170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:26:10.435549 4178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:10.435781 4170 server_base.cc:1034] running on GCE node
W20260928 20:26:10.435557 4175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:10.435582 4176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:10.436434 4170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:26:10.436726 4170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:26:10.437914 4170 hybrid_clock.cc:648] HybridClock initialized: now 1790627170437893 us; error 37 us; skew 500 ppm
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:10.440685 4170 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:26:10.441233 4170 webserver.cc:460] Webserver started at http://127.24.186.193:36213/ using document root <none> and password file <none>
I20260928 20:26:10.441486 4170 fs_manager.cc:359] Metadata directory not provided
I20260928 20:26:10.441556 4170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:26:10.441776 4170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:10.455199 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "98a77205e0564cbc938393f5f77489d2"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b8a0934e6acb599c98fcc972f01c154c"
server_key_iv: "5675cf3750a92e3e998e0ce9b19f90bb"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.455788 4170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "98a77205e0564cbc938393f5f77489d2"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b8a0934e6acb599c98fcc972f01c154c"
server_key_iv: "5675cf3750a92e3e998e0ce9b19f90bb"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.458009 4170 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 20:26:10.474647 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.475214 4170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 20:26:10.475394 4170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "98a77205e0564cbc938393f5f77489d2"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b8a0934e6acb599c98fcc972f01c154c"
server_key_iv: "5675cf3750a92e3e998e0ce9b19f90bb"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.475555 4170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:26:10.513370 4170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:26:10.515866 4170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:26:10.516023 4170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:26:10.516289 4170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:26:10.516672 4170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:26:10.516727 4170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.516767 4170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:26:10.516791 4170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.526937 4170 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:34483
I20260928 20:26:10.527442 4170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:26:10.527922 4300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:34483 every 8 connection(s)
I20260928 20:26:10.536103 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 4170
I20260928 20:26:10.536232 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:26:10.536432 25323 external_mini_cluster.cc:1428] Setting key 928ab96440e173b6b2d6e358da363f66
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:26:10.546808 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:10.529005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:46757 (local address 127.24.186.193:38885)
0928 20:26:10.529963 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:10.530019 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:10.530673 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:10.531205 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:10.531210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:10.531465 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:26:10.531775 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:10.531783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.533116 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.533117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:10.539729 (+ 6612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:10.539736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.540383 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.540384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:10.540555 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:10.541295 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:26:10.541305 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:26:10.543009 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:26:10.545089 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:10.545097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:10.545099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:10.545408 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:10.545634 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:10.545635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:10.545635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:10.545769 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:10.546096 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:26:10.546103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:26:10.546273 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 20:26:10.546386 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 20:26:10.546557 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":99,"threads_started":1}
I20260928 20:26:10.547472 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:10.529016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:38885 (local address 127.24.186.254:46757)
0928 20:26:10.530944 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:10.530947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:10.530956 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:26:10.530983 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:10.530984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:10.530998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:10.531069 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:10.531991 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.532955 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.539894 (+ 6939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.540223 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.540299 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:10.543128 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:10.543129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:10.543130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:10.543138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:10.544981 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:10.545500 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:10.545501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:10.545502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:10.545567 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:10.545842 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:10.545843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:10.545843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:10.546002 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:10.546067 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:10.547272 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0928 20:26:10.547370 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1822,"thread_start_us":90,"threads_started":1}
I20260928 20:26:10.548440 4301 heartbeater.cc:347] Connected to a master server at 127.24.186.254:46757
I20260928 20:26:10.548563 4301 heartbeater.cc:464] Registering TS with master...
I20260928 20:26:10.548785 4301 heartbeater.cc:511] Master 127.24.186.254:46757 requested a full tablet report, sending...
I20260928 20:26:10.549363 4089 ts_manager.cc:194] Registered new tserver with Master: 98a77205e0564cbc938393f5f77489d2 (127.24.186.193:34483)
I20260928 20:26:10.550285 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:38885
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:26:10.575392 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:50289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:26:10.705170 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:26:10.705471 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:26:10.705509 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:26:10.707345 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:26:10.707417 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:26:10.707463 4307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:26:10.710278 4307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.186.212:50289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:26:10.710588 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:26:10.710901 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:26:10.718783 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:10.719267 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:10.721102 4307 server_base.cc:1034] running on GCE node
W20260928 20:26:10.724047 4317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:10.724439 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:26:10.724764 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:26:10.725950 4307 hybrid_clock.cc:648] HybridClock initialized: now 1790627170725931 us; error 36 us; skew 500 ppm
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:10.729576 4307 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:26:10.730129 4307 webserver.cc:460] Webserver started at http://127.24.186.194:42369/ using document root <none> and password file <none>
I20260928 20:26:10.730389 4307 fs_manager.cc:359] Metadata directory not provided
I20260928 20:26:10.730437 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:26:10.730563 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:10.751925 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "77ea1afaad564c9fb64fb02fe890d415"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b1847e225665d63d7aa778083bf54af0"
server_key_iv: "1f420129eaee8a49dd8bec342e755975"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.752602 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "77ea1afaad564c9fb64fb02fe890d415"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b1847e225665d63d7aa778083bf54af0"
server_key_iv: "1f420129eaee8a49dd8bec342e755975"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.754958 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:10.774147 4324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.774353 4307 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 20:26:10.774457 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "77ea1afaad564c9fb64fb02fe890d415"
format_stamp: "Formatted at 2026-09-28 20:26:10 on dist-test-slave-k477"
server_key: "b1847e225665d63d7aa778083bf54af0"
server_key_iv: "1f420129eaee8a49dd8bec342e755975"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:10.774567 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:26:10.792701 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:26:10.795053 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:26:10.795200 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:26:10.795398 4307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:26:10.795765 4307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:26:10.795827 4307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.795864 4307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:26:10.795885 4307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:10.804662 4307 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:45465
I20260928 20:26:10.805075 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:26:10.808761 4437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:45465 every 8 connection(s)
I20260928 20:26:10.813838 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 4307
I20260928 20:26:10.813989 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:26:10.814204 25323 external_mini_cluster.cc:1428] Setting key 9bae54087c4ffc17508d522211df60da
Sep 28 20:26:10 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:26:10.831012 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:10.809656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:48961 (local address 127.24.186.254:46757)
0928 20:26:10.809710 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:10.809712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:10.810427 (+ 715us) server_negotiation.cc:407] Connection header received
0928 20:26:10.811674 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:10.811676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:10.811696 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:10.811728 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:10.812649 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.813466 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.824351 (+ 10885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.824692 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.824746 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:10.827514 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:10.827517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:10.827517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:10.827526 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:10.829389 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:10.829940 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:10.829942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:10.829942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:10.829998 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:10.830276 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:10.830277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:10.830277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:10.830412 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:10.830451 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:10.830816 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 20:26:10.830910 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:26:10.831534 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:10.809522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:46757 (local address 127.24.186.194:48961)
0928 20:26:10.810283 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:10.810330 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:10.811526 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:10.811971 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:10.811976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:10.812250 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:26:10.812535 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:10.812541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.822755 (+ 10214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.822756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:10.824197 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:10.824204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:10.824860 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:10.824861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:10.825025 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:10.825667 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:26:10.825679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:26:10.827377 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:26:10.829502 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:10.829512 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:10.829515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:10.829851 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:10.830077 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:10.830078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:10.830078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:10.830206 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:10.830493 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:26:10.830498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:26:10.830669 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 20:26:10.831133 (+ 464us) client_negotiation.cc:240] Negotiation successful
0928 20:26:10.831300 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":537,"threads_started":1}
I20260928 20:26:10.832073 4438 heartbeater.cc:347] Connected to a master server at 127.24.186.254:46757
I20260928 20:26:10.832176 4438 heartbeater.cc:464] Registering TS with master...
I20260928 20:26:10.832396 4438 heartbeater.cc:511] Master 127.24.186.254:46757 requested a full tablet report, sending...
I20260928 20:26:10.832782 4089 ts_manager.cc:194] Registered new tserver with Master: 77ea1afaad564c9fb64fb02fe890d415 (127.24.186.194:45465)
I20260928 20:26:10.833281 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:48961
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:26:10.860097 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:50289/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:26:11.000128 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:26:11.000455 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:26:11.000537 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:26:11.001894 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:26:11.001994 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:26:11.002044 4443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:26:11.004681 4443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.186.212:50289/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:46757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:26:11.005018 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:26:11.005306 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:26:11.013741 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:11.014029 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:11.015383 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:11.015482 4443 server_base.cc:1034] running on GCE node
I20260928 20:26:11.015944 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:26:11.016269 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:26:11.017491 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790627171017442 us; error 54 us; skew 500 ppm
Sep 28 20:26:11 dist-test-slave-k477 krb5kdc[3613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627171, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:11.021304 4443 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:26:11.022038 4443 webserver.cc:460] Webserver started at http://127.24.186.195:36987/ using document root <none> and password file <none>
I20260928 20:26:11.022329 4443 fs_manager.cc:359] Metadata directory not provided
I20260928 20:26:11.022394 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:26:11.022535 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:26:11 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627171, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:11.043519 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "82f2648c3daa42a49d339608361001b6"
format_stamp: "Formatted at 2026-09-28 20:26:11 on dist-test-slave-k477"
server_key: "b95855a76aec248b7b4ae4bdc7c7c9f1"
server_key_iv: "3acd759e4d5e5b3356cd5bade8102834"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:11.044034 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "82f2648c3daa42a49d339608361001b6"
format_stamp: "Formatted at 2026-09-28 20:26:11 on dist-test-slave-k477"
server_key: "b95855a76aec248b7b4ae4bdc7c7c9f1"
server_key_iv: "3acd759e4d5e5b3356cd5bade8102834"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:11.046214 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 20:26:11.083156 4459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:11.083360 4443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:26:11.083457 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "82f2648c3daa42a49d339608361001b6"
format_stamp: "Formatted at 2026-09-28 20:26:11 on dist-test-slave-k477"
server_key: "b95855a76aec248b7b4ae4bdc7c7c9f1"
server_key_iv: "3acd759e4d5e5b3356cd5bade8102834"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:11.083568 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:26:11.112221 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:26:11.114663 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:26:11.114780 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:26:11.115037 4443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:26:11.115443 4443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:26:11.115486 4443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:11.115510 4443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:26:11.115523 4443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:11.131052 4443 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:44653
I20260928 20:26:11.131193 4572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:44653 every 8 connection(s)
I20260928 20:26:11.131524 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:26:11.132041 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 4443
I20260928 20:26:11.132151 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:26:11.132352 25323 external_mini_cluster.cc:1428] Setting key 93727f8d40c60ea15160ce97edede3db
Sep 28 20:26:11 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627171, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:26:11.142575 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.132491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:50361 (local address 127.24.186.254:46757)
0928 20:26:11.132871 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.132873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.132934 (+ 61us) server_negotiation.cc:407] Connection header received
0928 20:26:11.133698 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.133699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.133711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.133755 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:11.134401 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.134912 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.136035 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.136251 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.136339 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.138735 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:11.138737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:11.138737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:11.138745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:11.141080 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:11.141581 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:11.141582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:11.141582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:11.141639 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:11.141897 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:11.141898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:11.141898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:11.142027 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:11.142096 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.142390 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.142478 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260928 20:26:11.142860 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.132456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:46757 (local address 127.24.186.195:50361)
0928 20:26:11.132758 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.132804 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.133598 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.133828 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.133832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.134085 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:26:11.134310 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.134314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.135010 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.135011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.135937 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.135940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.136349 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.136350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.136519 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.137134 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:26:11.137143 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:26:11.138634 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:26:11.141218 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:11.141229 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:11.141232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:11.141488 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:11.141716 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:26:11.141717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:26:11.141717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:26:11.141827 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:26:11.142098 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:26:11.142105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:26:11.142254 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.142357 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.142515 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":209,"threads_started":1}
I20260928 20:26:11.143486 4573 heartbeater.cc:347] Connected to a master server at 127.24.186.254:46757
I20260928 20:26:11.143697 4573 heartbeater.cc:464] Registering TS with master...
I20260928 20:26:11.144009 4573 heartbeater.cc:511] Master 127.24.186.254:46757 requested a full tablet report, sending...
I20260928 20:26:11.144446 4089 ts_manager.cc:194] Registered new tserver with Master: 82f2648c3daa42a49d339608361001b6 (127.24.186.195:44653)
I20260928 20:26:11.144923 4089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:50361
I20260928 20:26:11.155014 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:26:11.165055 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.156838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.24.186.254:46757)
0928 20:26:11.156908 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.156911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.157146 (+ 235us) server_negotiation.cc:407] Connection header received
0928 20:26:11.157186 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.157187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.157198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.157278 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:11.158065 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.158790 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.159890 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.160208 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.160290 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.161928 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:11.161930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:11.161931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:11.161939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:11.163695 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:11.164179 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:11.164180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:11.164180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:11.164242 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:11.164447 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:11.164448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:11.164448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:11.164577 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:11.164653 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.164873 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.164956 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:26:11.167244 4089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:26:11.169142 4089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:26:11.182498 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.175965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.24.186.194:45465)
0928 20:26:11.176344 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.176349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.176753 (+ 404us) server_negotiation.cc:407] Connection header received
0928 20:26:11.176808 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.176810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.176826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.176961 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:26:11.177223 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.178420 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.180447 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.181397 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.181525 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.181601 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.182311 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.182366 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":188,"threads_started":1}
I20260928 20:26:11.182964 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.175428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:45465 (local address 127.0.0.1:51866)
0928 20:26:11.176568 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.176579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.176694 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.177089 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.177090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.177092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:26:11.177138 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.177140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.178509 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.178510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.180279 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.180282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.181642 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.181643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.181739 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.181754 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.181829 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.181887 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":171,"threads_started":1}
I20260928 20:26:11.183727 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.175419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:34483 (local address 127.0.0.1:38578)
0928 20:26:11.175734 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.175779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.175964 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.176490 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.176494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.176496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:26:11.176583 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.176585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.178203 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.178204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.180110 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.180113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.183498 (+ 3385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.183499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.183573 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.183579 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.183612 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.183656 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":187,"threads_started":1}
I20260928 20:26:11.184106 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.175992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.24.186.193:34483)
0928 20:26:11.176182 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.176187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.176198 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:26:11.176235 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.176237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.176254 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.176424 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:26:11.176950 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.178100 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.180209 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.181298 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.183863 (+ 2565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.183940 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.183969 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.184016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260928 20:26:11.185637 4235 tablet_service.cc:1467] Processing CreateTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 (DEFAULT_TABLE table=test-table [id=fc3dd2b81b97420b88a0f39dc3a65a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:26:11.187050 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb13e490564f4ffba8be3dfa4625e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:26:11.187852 4372 tablet_service.cc:1467] Processing CreateTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 (DEFAULT_TABLE table=test-table [id=fc3dd2b81b97420b88a0f39dc3a65a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:26:11.188268 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb13e490564f4ffba8be3dfa4625e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:26:11.196442 4591 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Bootstrap starting.
I20260928 20:26:11.197890 4591 tablet_bootstrap.cc:654] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Neither blocks nor log segments found. Creating new log.
I20260928 20:26:11.198529 4591 log.cc:824] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Log is configured to *not* fsync() on all Append() calls
I20260928 20:26:11.199286 4592 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Bootstrap starting.
I20260928 20:26:11.199975 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.176174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44653 (local address 127.0.0.1:37826)
0928 20:26:11.177097 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.177106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.177195 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.177538 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.177539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.177541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:26:11.177577 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.177578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.196651 (+ 19073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.196652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.197875 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.197878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.199332 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.199334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.199417 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.199422 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.199618 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.199730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":175,"threads_started":1}
I20260928 20:26:11.200398 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.176680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37826 (local address 127.24.186.195:44653)
0928 20:26:11.177069 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.177074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.177252 (+ 178us) server_negotiation.cc:407] Connection header received
0928 20:26:11.177307 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.177309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.177324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.177442 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:26:11.177652 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.196172 (+ 18520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.198060 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.198932 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.199205 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.199280 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.200223 (+ 943us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.200280 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":102,"threads_started":1}
I20260928 20:26:11.201170 4507 tablet_service.cc:1467] Processing CreateTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 (DEFAULT_TABLE table=test-table [id=fc3dd2b81b97420b88a0f39dc3a65a30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:26:11.201574 4507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb13e490564f4ffba8be3dfa4625e5e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:26:11.201633 4592 tablet_bootstrap.cc:654] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Neither blocks nor log segments found. Creating new log.
I20260928 20:26:11.202406 4592 log.cc:824] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Log is configured to *not* fsync() on all Append() calls
I20260928 20:26:11.203197 4591 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: No bootstrap required, opened a new log
I20260928 20:26:11.203275 4591 ts_tablet_manager.cc:1397] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 20:26:11.206030 4591 raft_consensus.cc:348] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.206220 4591 raft_consensus.cc:374] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:26:11.206259 4591 raft_consensus.cc:729] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ea1afaad564c9fb64fb02fe890d415, State: Initialized, Role: FOLLOWER
I20260928 20:26:11.206389 4591 consensus_queue.cc:260] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.206720 4591 ts_tablet_manager.cc:1428] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:26:11.206863 4438 heartbeater.cc:503] Master 127.24.186.254:46757 was elected leader, sending a full tablet report...
I20260928 20:26:11.208004 4592 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: No bootstrap required, opened a new log
I20260928 20:26:11.208088 4592 ts_tablet_manager.cc:1397] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260928 20:26:11.208519 4595 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Bootstrap starting.
I20260928 20:26:11.210104 4595 tablet_bootstrap.cc:654] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Neither blocks nor log segments found. Creating new log.
I20260928 20:26:11.210688 4595 log.cc:824] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:26:11.211400 4592 raft_consensus.cc:348] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.211592 4592 raft_consensus.cc:374] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:26:11.211620 4595 tablet_bootstrap.cc:492] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: No bootstrap required, opened a new log
I20260928 20:26:11.211709 4595 ts_tablet_manager.cc:1397] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:26:11.211627 4592 raft_consensus.cc:729] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a77205e0564cbc938393f5f77489d2, State: Initialized, Role: FOLLOWER
I20260928 20:26:11.211974 4592 consensus_queue.cc:260] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.212832 4592 ts_tablet_manager.cc:1428] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 20:26:11.213023 4301 heartbeater.cc:503] Master 127.24.186.254:46757 was elected leader, sending a full tablet report...
I20260928 20:26:11.214574 4595 raft_consensus.cc:348] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.214804 4595 raft_consensus.cc:374] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:26:11.214850 4595 raft_consensus.cc:729] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f2648c3daa42a49d339608361001b6, State: Initialized, Role: FOLLOWER
I20260928 20:26:11.216411 4595 consensus_queue.cc:260] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.216918 4595 ts_tablet_manager.cc:1428] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 20:26:11.216980 4573 heartbeater.cc:503] Master 127.24.186.254:46757 was elected leader, sending a full tablet report...
W20260928 20:26:11.279181 4302 tablet.cc:2369] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:26:11.310704 4439 tablet.cc:2369] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 20:26:11 dist-test-slave-k477 krb5kdc[3613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627170, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
W20260928 20:26:11.382443 4574 tablet.cc:2369] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:26:11.405819 4598 raft_consensus.cc:482] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:26:11.406008 4598 raft_consensus.cc:504] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.406418 4598 leader_election.cc:288] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f2648c3daa42a49d339608361001b6 (127.24.186.195:44653), 77ea1afaad564c9fb64fb02fe890d415 (127.24.186.194:45465)
I20260928 20:26:11.413826 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.407073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:53441 (local address 127.24.186.194:45465)
0928 20:26:11.407118 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.407120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.407866 (+ 746us) server_negotiation.cc:407] Connection header received
0928 20:26:11.407969 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.407970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.407984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.408013 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:26:11.408276 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.408960 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.412510 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.413080 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.413133 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.413141 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.413616 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.413743 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:26:11.414397 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.407008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:45465 (local address 127.24.186.193:53441)
0928 20:26:11.407734 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.407744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.407892 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.408076 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.408076 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.408078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:26:11.408195 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.408196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.409065 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.409066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.410440 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.410443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.413210 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.413211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.413326 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.413331 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.414200 (+ 869us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.414255 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":55,"thread_start_us":101,"threads_started":1}
I20260928 20:26:11.414826 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.406843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52223 (local address 127.24.186.195:44653)
0928 20:26:11.406888 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.406890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.407583 (+ 693us) server_negotiation.cc:407] Connection header received
0928 20:26:11.407695 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.407697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.407708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.407742 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:26:11.408130 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.408916 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.411796 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.412368 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.412429 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.412439 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.414697 (+ 2258us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.414758 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:26:11.415644 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb13e490564f4ffba8be3dfa4625e5e3" candidate_uuid: "98a77205e0564cbc938393f5f77489d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ea1afaad564c9fb64fb02fe890d415" is_pre_election: true
I20260928 20:26:11.415843 4392 raft_consensus.cc:2454] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a77205e0564cbc938393f5f77489d2 in term 0.
I20260928 20:26:11.416368 4188 leader_election.cc:302] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ea1afaad564c9fb64fb02fe890d415, 98a77205e0564cbc938393f5f77489d2; no voters:
I20260928 20:26:11.416525 4598 raft_consensus.cc:2781] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:26:11.416569 4598 raft_consensus.cc:482] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:26:11.416590 4598 raft_consensus.cc:3037] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:26:11.417745 4598 raft_consensus.cc:504] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.417886 4598 leader_election.cc:288] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [CANDIDATE]: Term 1 election: Requested vote from peers 82f2648c3daa42a49d339608361001b6 (127.24.186.195:44653), 77ea1afaad564c9fb64fb02fe890d415 (127.24.186.194:45465)
I20260928 20:26:11.418246 4392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb13e490564f4ffba8be3dfa4625e5e3" candidate_uuid: "98a77205e0564cbc938393f5f77489d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ea1afaad564c9fb64fb02fe890d415"
I20260928 20:26:11.418329 4392 raft_consensus.cc:3037] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:26:11.419477 4392 raft_consensus.cc:2454] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a77205e0564cbc938393f5f77489d2 in term 1.
I20260928 20:26:11.419951 4188 leader_election.cc:302] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ea1afaad564c9fb64fb02fe890d415, 98a77205e0564cbc938393f5f77489d2; no voters:
I20260928 20:26:11.415230 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.406625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:44653 (local address 127.24.186.193:52223)
0928 20:26:11.407524 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.407536 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:26:11.407613 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:26:11.407858 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:26:11.407859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:26:11.407861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:26:11.408003 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.408005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.409072 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.409073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.411524 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:26:11.411527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.413419 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.413420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:26:11.413517 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.413521 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:26:11.415076 (+ 1555us) client_negotiation.cc:240] Negotiation successful
0928 20:26:11.415127 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":1174,"thread_start_us":119,"threads_started":1}
I20260928 20:26:11.420617 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb13e490564f4ffba8be3dfa4625e5e3" candidate_uuid: "98a77205e0564cbc938393f5f77489d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f2648c3daa42a49d339608361001b6" is_pre_election: true
I20260928 20:26:11.420816 4527 raft_consensus.cc:2454] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98a77205e0564cbc938393f5f77489d2 in term 0.
I20260928 20:26:11.420966 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb13e490564f4ffba8be3dfa4625e5e3" candidate_uuid: "98a77205e0564cbc938393f5f77489d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f2648c3daa42a49d339608361001b6"
I20260928 20:26:11.421034 4526 raft_consensus.cc:3037] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:26:11.422322 4526 raft_consensus.cc:2454] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98a77205e0564cbc938393f5f77489d2 in term 1.
I20260928 20:26:11.420096 4598 raft_consensus.cc:2781] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:26:11.422876 4598 raft_consensus.cc:686] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [term 1 LEADER]: Becoming Leader. State: Replica: 98a77205e0564cbc938393f5f77489d2, State: Running, Role: LEADER
I20260928 20:26:11.422983 4598 consensus_queue.cc:237] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } }
I20260928 20:26:11.423913 4089 catalog_manager.cc:5495] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98a77205e0564cbc938393f5f77489d2 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "98a77205e0564cbc938393f5f77489d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98a77205e0564cbc938393f5f77489d2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 34483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 } health_report { overall_health: UNKNOWN } } }
I20260928 20:26:11.468773 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.465308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.24.186.193:34483)
0928 20:26:11.465382 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:11.465385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:11.465398 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:26:11.465438 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:11.465439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:11.465464 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:11.465544 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:26:11.465774 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.466510 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.467599 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:11.467897 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:11.467951 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:11.468134 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:26:11.468272 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:26:11.468339 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:11.468625 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:26:11.468648 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 20:26:11.471149 4526 raft_consensus.cc:1261] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 1 FOLLOWER]: Refusing update from remote peer 98a77205e0564cbc938393f5f77489d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:26:11.471175 4392 raft_consensus.cc:1261] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 1 FOLLOWER]: Refusing update from remote peer 98a77205e0564cbc938393f5f77489d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:26:11.471555 4602 consensus_queue.cc:1035] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f2648c3daa42a49d339608361001b6" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:26:11.471727 4598 consensus_queue.cc:1035] T cb13e490564f4ffba8be3dfa4625e5e3 P 98a77205e0564cbc938393f5f77489d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ea1afaad564c9fb64fb02fe890d415" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:26:11.474447 4605 mvcc.cc:205] Tried to move back new op lower bound from 7334408894342987776 to 7334408894149115904. Current Snapshot: MvccSnapshot[applied={T|T < 7334408894342987776}]
I20260928 20:26:11.479292 4089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:26:11.479403 4089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:26:11.479919 4609 mvcc.cc:205] Tried to move back new op lower bound from 7334408894342987776 to 7334408894149115904. Current Snapshot: MvccSnapshot[applied={T|T < 7334408894342987776}]
I20260928 20:26:11.480305 4089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2ab2724669d4c7aa34318b5b60aa905: Sending DeleteTablet for 3 replicas of tablet cb13e490564f4ffba8be3dfa4625e5e3
I20260928 20:26:11.480502 4606 mvcc.cc:205] Tried to move back new op lower bound from 7334408894342987776 to 7334408894149115904. Current Snapshot: MvccSnapshot[applied={T|T < 7334408894342987776}]
I20260928 20:26:11.480767 4235 tablet_service.cc:1514] Processing DeleteTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:26:11 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38578
I20260928 20:26:11.481042 4372 tablet_service.cc:1514] Processing DeleteTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:26:11 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:51866
I20260928 20:26:11.481086 4507 tablet_service.cc:1514] Processing DeleteTablet for tablet cb13e490564f4ffba8be3dfa4625e5e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:26:11 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:37826
I20260928 20:26:11.481593 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 4170
I20260928 20:26:11.486311 4616 tablet_replica.cc:330] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: stopping tablet replica
I20260928 20:26:11.486493 4616 raft_consensus.cc:2229] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:26:11.486706 4616 raft_consensus.cc:2258] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:26:11.487275 4616 ts_tablet_manager.cc:1905] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:26:11.487955 4615 tablet_replica.cc:330] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: stopping tablet replica
I20260928 20:26:11.488158 4615 raft_consensus.cc:2229] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:26:11.489042 4615 raft_consensus.cc:2258] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:26:11.489642 4615 ts_tablet_manager.cc:1905] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:26:11.494175 4615 ts_tablet_manager.cc:1918] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:26:11.494300 4616 ts_tablet_manager.cc:1918] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:26:11.494416 4616 log.cc:1196] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/cb13e490564f4ffba8be3dfa4625e5e3
I20260928 20:26:11.494740 4616 ts_tablet_manager.cc:1939] T cb13e490564f4ffba8be3dfa4625e5e3 P 77ea1afaad564c9fb64fb02fe890d415: Deleting consensus metadata
I20260928 20:26:11.495286 4075 catalog_manager.cc:4841] TS 77ea1afaad564c9fb64fb02fe890d415 (127.24.186.194:45465): tablet cb13e490564f4ffba8be3dfa4625e5e3 (table test-table [id=fc3dd2b81b97420b88a0f39dc3a65a30]) successfully deleted
W20260928 20:26:11.495836 4074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:34483 (error 108)
I20260928 20:26:11.495862 4615 log.cc:1196] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/cb13e490564f4ffba8be3dfa4625e5e3
I20260928 20:26:11.496162 4615 ts_tablet_manager.cc:1939] T cb13e490564f4ffba8be3dfa4625e5e3 P 82f2648c3daa42a49d339608361001b6: Deleting consensus metadata
I20260928 20:26:11.496682 4074 catalog_manager.cc:4841] TS 82f2648c3daa42a49d339608361001b6 (127.24.186.195:44653): tablet cb13e490564f4ffba8be3dfa4625e5e3 (table test-table [id=fc3dd2b81b97420b88a0f39dc3a65a30]) successfully deleted
I20260928 20:26:11.496744 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:11.496634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:34483 (local address 127.0.0.1:38594)
0928 20:26:11.496673 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 20:26:11.496711 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 20:26:11.496836 4074 catalog_manager.cc:4584] TS 98a77205e0564cbc938393f5f77489d2 (127.24.186.193:34483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb13e490564f4ffba8be3dfa4625e5e3: Network error: Client connection negotiation failed: client connection to 127.24.186.193:34483: connect: Connection refused (error 111)
I20260928 20:26:11.496944 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 4307
I20260928 20:26:11.506793 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 4443
I20260928 20:26:11.513331 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 4057
I20260928 20:26:11.520313 25323 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 20:26:11.865993 25323 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 20:26:11.866125 25323 mini_ranger.cc:66] Stopping Ranger...
I20260928 20:26:12.211585 25323 mini_ranger.cc:68] Stopped Ranger
2026-09-28 20:26:12.211 UTC [3643] LOG: received smart shutdown request
2026-09-28 20:26:12.215 UTC [3643] LOG: background worker "logical replication launcher" (PID 3652) exited with exit code 1
2026-09-28 20:26:12.215 UTC [3647] LOG: shutting down
2026-09-28 20:26:12.406 UTC [3643] LOG: database system is shut down
2026-09-28T20:26:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51141 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:26:12 dist-test-slave-k477 krb5kdc[4626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:26:12 dist-test-slave-k477 krb5kdc[4626](info): set up 2 sockets
Sep 28 20:26:12 dist-test-slave-k477 krb5kdc[4626](info): commencing operation
krb5kdc: starting...
W20260928 20:26:13.604581 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:26:13 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:26:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangeradmin_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangeradmin_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangerlookup_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangerlookup_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
I20260928 20:26:13.787767 25323 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9jCLVo/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/postgres -l logfile start
2026-09-28 20:26:16.612 UTC [4656] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 20:26:16.612 UTC [4656] LOG: listening on IPv4 address "127.24.186.212", port 33909
2026-09-28 20:26:16.616 UTC [4656] LOG: listening on Unix socket "/tmp/.s.PGSQL.33909"
2026-09-28 20:26:16.645 UTC [4659] LOG: database system was shut down at 2026-09-28 20:26:14 UTC
2026-09-28 20:26:16.649 UTC [4656] LOG: database system is ready to accept connections
I20260928 20:26:17.624303 25323 mini_postgres.cc:96] Postgres bound to 33909
2026-09-28 20:26:17.630 UTC [4667] FATAL: database "slave" does not exist
127.24.186.212:33909 - accepting connections
I20260928 20:26:17.631572 25323 mini_ranger.cc:161] Starting Ranger...
I20260928 20:26:17.648000 25323 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 20:26:18.168392 25323 mini_ranger.cc:87] Created ranger Postgres database
I20260928 20:26:18.168483 25323 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin
2026-09-28 20:26:18,461 [I] DB FLAVOR :POSTGRES
2026-09-28 20:26:18,461 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:26:18,461 [I] Checking connection..
2026-09-28 20:26:18,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 20:26:18,719 [I] Checking connection passed.
2026-09-28 20:26:18,719 [I] --------- Verifying version history table ---------
2026-09-28 20:26:18,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:26:18,978 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:26:18,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:26:19,228 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 20:26:19,228 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 20:26:19,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 20:26:19.475 UTC [4731] WARNING: there is no transaction in progress
2026-09-28 20:26:19,482 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 20:26:19,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 20:26:19,751 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 20:26:19,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:26:19,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 20:26:20,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 20:26:20,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-k477.c.gcp-upstream.internal','N') ;"
2026-09-28 20:26:20,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:26:20,817 [I] Table x_portal_user does not exist in database ranger
2026-09-28 20:26:20,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:26:21,077 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 20:26:21,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:26:21,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:26:21,591 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 20:26:21,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 20:26:22.643 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.663 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.682 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.700 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.719 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.737 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.804 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.813 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.830 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.844 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.857 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.873 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.884 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.891 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.899 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:22.908 UTC [4839] WARNING: there is no transaction in progress
2026-09-28 20:26:23,603 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 20:26:23,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 20:26:23,861 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 20:26:23,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 20:26:24,144 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 20:26:24,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 20:26:24,435 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 20:26:24,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 20:26:24,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 20:26:24,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-k477.c.gcp-upstream.internal';"
2026-09-28 20:26:25,229 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 20:26:25,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 20:26:25,502 [I] DB_PATCHES have already been applied
I20260928 20:26:25.508375 25323 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9jCLVo/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 20:26:25.508453 25323 mini_ranger.cc:193] Using host: 127.24.186.212
I20260928 20:26:25.511905 25323 mini_ranger.cc:227] Ranger admin URL: http://127.24.186.212:60933
Sep 28, 2026 8:26:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:26:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:26:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:26:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:26:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:26:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:26:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 20:26:26.666627 25323 mini_ranger.cc:160] Time spent starting Ranger: real 9.035s user 0.000s sys 0.012s
Sep 28, 2026 8:26:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:26:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:26:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:26:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:26:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 20:26:49.401963 25323 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.24.186.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.24.186.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangerkms_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.24.186.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/rangerkms_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.24.186.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.24.186.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.24.186.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/HTTP_127.24.186.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 20:26:49 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 20:26:49.493116 25323 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 20:26:49.505013 25323 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 20:26:50.531337 25323 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 20:26:50.531430 25323 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms
I20260928 20:26:50.531452 25323 mini_ranger_kms.cc:227] Using postgres at 127.24.186.212:33909
2026-09-28 20:26:50,711 [I] DB FLAVOR :POSTGRES
2026-09-28 20:26:50,711 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 20:26:50,712 [I] Checking connection
2026-09-28 20:26:50,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 20:26:50,981 [I] connection success
2026-09-28 20:26:50,981 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 20:26:50,981 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 20:26:50,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 20:26:51,247 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 20:26:51,247 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 20:26:51,247 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 20:26:51,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.24.186.212:33909/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 20:26:51,522 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 20:26:52.099268 25323 mini_ranger_kms.cc:310] Created kms service
I20260928 20:26:52.378608 25323 mini_ranger_kms.cc:326] Created kudu user
I20260928 20:26:52.424481 25323 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 20:26:52 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627209, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:52.860771 25323 mini_ranger_kms.cc:384] Added ranger policy
I20260928 20:26:52.861001 25323 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms:/tmp/dist-test-task9jCLVo/build/release/bin/postgresql.jar:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9jCLVo/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 20:26:52.861037 25323 mini_ranger_kms.cc:240] Using host: 127.24.186.212
I20260928 20:26:52.865232 25323 mini_ranger_kms.cc:276] Ranger KMS PID: 5020
I20260928 20:26:52.865317 25323 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.24.186.212:47063
Sep 28, 2026 8:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:26:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:26:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:26:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 20:26:55.157402 25323 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.664s user 0.013s sys 0.147s
I20260928 20:26:55.157519 25323 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 20:26:55.157570 25323 mini_ranger_kms.cc:401] 127.24.186.212:47063/kms/v1/keys
Sep 28 20:26:55 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790627215, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 20:26:55 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627215, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
Sep 28 20:26:56 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627215, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.24.186.212@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
Sep 28 20:26:58 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:26:58.900377 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33123
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33123
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:26:59.009414 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:26:59.009702 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:26:59.009742 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:26:59.011029 5070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:26:59.011106 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:26:59.011130 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:26:59.011161 5070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:26:59.011193 5070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:26:59.012920 5070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
--ranger_kms_url=127.24.186.212:47063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33123
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:26:59.013236 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:26:59.013477 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:26:59.019095 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:26:59.019095 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:59.019178 5070 server_base.cc:1034] running on GCE node
W20260928 20:26:59.019299 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:26:59.019680 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:26:59.020114 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:26:59.021302 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790627219021278 us; error 41 us; skew 500 ppm
Sep 28 20:26:59 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:26:59.024426 5070 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:26:59.025104 5070 webserver.cc:460] Webserver started at http://127.24.186.254:34957/ using document root <none> and password file <none>
I20260928 20:26:59.025388 5070 fs_manager.cc:359] Metadata directory not provided
I20260928 20:26:59.025458 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:26:59.025745 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:26:59 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627219, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:26:59.086192 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "cae45b5f68254d98a07397a08d658bd3"
format_stamp: "Formatted at 2026-09-28 20:26:59 on dist-test-slave-k477"
server_key: "d9dd9188ffb72c591cc09df650f88e68"
server_key_iv: "08ab1b7212ab47eee4d8aef7c3df8582"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:59.086714 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "cae45b5f68254d98a07397a08d658bd3"
format_stamp: "Formatted at 2026-09-28 20:26:59 on dist-test-slave-k477"
server_key: "d9dd9188ffb72c591cc09df650f88e68"
server_key_iv: "08ab1b7212ab47eee4d8aef7c3df8582"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:59.088945 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 20:26:59.101233 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:26:59.101450 5070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:26:59.101547 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "cae45b5f68254d98a07397a08d658bd3"
format_stamp: "Formatted at 2026-09-28 20:26:59 on dist-test-slave-k477"
server_key: "d9dd9188ffb72c591cc09df650f88e68"
server_key_iv: "08ab1b7212ab47eee4d8aef7c3df8582"
server_key_version: "kuduclusterkey@0"
I20260928 20:26:59.101666 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:26:59.111922 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:26:59.114059 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:26:59.114234 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:26:59.119735 5070 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:33123
I20260928 20:26:59.119848 5137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:33123 every 8 connection(s)
I20260928 20:26:59.120209 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:26:59.120884 5138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:26:59.123206 5138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Bootstrap starting.
I20260928 20:26:59.123917 5138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Neither blocks nor log segments found. Creating new log.
I20260928 20:26:59.124222 5138 log.cc:824] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:26:59.124977 5138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: No bootstrap required, opened a new log
I20260928 20:26:59.127005 5138 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:26:59.127156 5138 raft_consensus.cc:374] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:26:59.127178 5138 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae45b5f68254d98a07397a08d658bd3, State: Initialized, Role: FOLLOWER
I20260928 20:26:59.127338 5138 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:26:59.127421 5138 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:26:59.127445 5138 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:26:59.127507 5138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:26:59.128156 5138 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:26:59.128242 5138 leader_election.cc:302] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae45b5f68254d98a07397a08d658bd3; no voters:
I20260928 20:26:59.128497 5138 leader_election.cc:288] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:26:59.128566 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5070
I20260928 20:26:59.128525 5141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:26:59.128676 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:26:59.128753 5141 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 LEADER]: Becoming Leader. State: Replica: cae45b5f68254d98a07397a08d658bd3, State: Running, Role: LEADER
I20260928 20:26:59.128846 25323 external_mini_cluster.cc:1428] Setting key f3f7bba2d59d067336eab7dc7ad2a442
I20260928 20:26:59.128844 5141 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:26:59.128868 5138 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:26:59.129570 5141 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae45b5f68254d98a07397a08d658bd3. Latest consensus state: current_term: 1 leader_uuid: "cae45b5f68254d98a07397a08d658bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } } }
I20260928 20:26:59.129753 5141 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 20:26:59.129799 5143 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cae45b5f68254d98a07397a08d658bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } } }
I20260928 20:26:59.129942 5143 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 20:26:59.130012 5147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:26:59.130726 5147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:26:59.131435 5138 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 20:26:59.135382 5159 mvcc.cc:205] Tried to move back new op lower bound from 7334409089564893184 to 7334409089552728064. Current Snapshot: MvccSnapshot[applied={T|T < 7334409089564893184}]
I20260928 20:26:59.136065 5147 catalog_manager.cc:1347] Generated new cluster ID: 2725db6bc60b4db1be97ab0a5958a6e9
I20260928 20:26:59.136123 5147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:26:59.142549 5147 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:26:59.143409 5147 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 20:26:59 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:26:59.148798 5147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Generated new TSK 0
I20260928 20:26:59.149058 5147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:26:59.153395 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:26:59.129659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.24.186.254:33123)
0928 20:26:59.135647 (+ 5988us) server_negotiation.cc:206] Beginning negotiation
0928 20:26:59.135662 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 20:26:59.135700 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:26:59.135872 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:26:59.135880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:26:59.135949 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:26:59.136128 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:26:59.142351 (+ 6223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:59.143900 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:59.145101 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:26:59.145735 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:26:59.145826 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:26:59.148990 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:26:59.148996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:26:59.148996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:26:59.149010 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:26:59.151171 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:59.151637 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:59.151642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:59.151642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:59.151749 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:26:59.151995 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:26:59.151996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:26:59.151996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:26:59.152306 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:26:59.152458 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 20:26:59.152984 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 20:26:59.153148 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5922,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:26:59.930819 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:00.053501 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:00.053759 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:00.053798 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:00.055226 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:00.055291 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:00.055331 5184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:00.058319 5184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:00.058635 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:00.058908 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:00.066601 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.066665 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.066970 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:00.067328 5184 server_base.cc:1034] running on GCE node
I20260928 20:27:00.067554 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:00.067786 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:00.069103 5184 hybrid_clock.cc:648] HybridClock initialized: now 1790627220069079 us; error 38 us; skew 500 ppm
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:00.071962 5184 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:00.072685 5184 webserver.cc:460] Webserver started at http://127.24.186.193:35175/ using document root <none> and password file <none>
I20260928 20:27:00.072947 5184 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:00.073019 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:00.073166 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:00.089502 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "b73457a4be4c4a809ff80609effcd61c"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "e4a0cb6032ddc0405223caac9af24e3b"
server_key_iv: "1cd903e98021756dd4b15535a2462365"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.089993 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "b73457a4be4c4a809ff80609effcd61c"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "e4a0cb6032ddc0405223caac9af24e3b"
server_key_iv: "1cd903e98021756dd4b15535a2462365"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.092150 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:27:00.108321 5200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.110445 5184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 20:27:00.110574 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b73457a4be4c4a809ff80609effcd61c"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "e4a0cb6032ddc0405223caac9af24e3b"
server_key_iv: "1cd903e98021756dd4b15535a2462365"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.110692 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:00.140256 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:00.142592 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:00.142777 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:00.143059 5184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:00.143453 5184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:00.143507 5184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.143575 5184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:00.143651 5184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.150646 5184 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:36545
I20260928 20:27:00.151242 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:00.158859 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5184
I20260928 20:27:00.158980 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:27:00.159165 25323 external_mini_cluster.cc:1428] Setting key ce8ae14a18f7ea6a7809e086b0d86411
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:00.153993 5314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:36545 every 8 connection(s)
I20260928 20:27:00.167760 5317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.154616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.193:40465)
0928 20:27:00.155207 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:00.155253 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:00.156411 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:00.156888 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:00.156894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:00.157163 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:00.157448 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.157454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.158485 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.158486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.160128 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.160131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.160680 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.160681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.160825 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.161471 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:00.161483 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:00.163366 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:00.165941 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.165949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.165952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.166288 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.166521 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.166522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.166522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.166667 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.166978 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:00.166984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:00.167223 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 20:27:00.167333 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:27:00.167509 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":89,"threads_started":1}
I20260928 20:27:00.168506 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.154923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:40465 (local address 127.24.186.254:33123)
0928 20:27:00.156629 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:00.156632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:00.156640 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:00.156669 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:00.156670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:00.156683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:00.156745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:00.157565 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.158373 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.160219 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.160539 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.160600 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.163490 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:00.163491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:00.163492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:00.163501 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:00.165824 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.166378 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.166380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.166380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.166451 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.166741 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.166742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.166742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.166890 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:00.166952 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:00.168251 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0928 20:27:00.168362 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1678,"thread_start_us":55,"threads_started":1}
I20260928 20:27:00.169346 5315 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:00.169459 5315 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:00.169663 5315 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:00.170226 5102 ts_manager.cc:194] Registered new tserver with Master: b73457a4be4c4a809ff80609effcd61c (127.24.186.193:36545)
I20260928 20:27:00.170866 5102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:40465
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:27:00.219544 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 20:27:00.342658 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:00.343078 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:00.343161 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:00.345193 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:00.345314 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:00.345413 5321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:00.347875 5321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:00.348369 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:00.348708 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:00.354794 5329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.354872 5328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.355188 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:00.355578 5321 server_base.cc:1034] running on GCE node
I20260928 20:27:00.355792 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:00.356096 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:00.357260 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790627220357241 us; error 36 us; skew 500 ppm
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:00.361457 5321 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:00.362072 5321 webserver.cc:460] Webserver started at http://127.24.186.194:37459/ using document root <none> and password file <none>
I20260928 20:27:00.362375 5321 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:00.362437 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:00.362612 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:00.377280 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "4d51731cb051487cafe8e40afed041bf"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "633a766531670dfd430528e3bbbd5b17"
server_key_iv: "ce59273865f7c3224f5f3fcd982edafd"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.377920 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "4d51731cb051487cafe8e40afed041bf"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "633a766531670dfd430528e3bbbd5b17"
server_key_iv: "ce59273865f7c3224f5f3fcd982edafd"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.380523 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:00.404534 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.404817 5321 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260928 20:27:00.404925 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "4d51731cb051487cafe8e40afed041bf"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "633a766531670dfd430528e3bbbd5b17"
server_key_iv: "ce59273865f7c3224f5f3fcd982edafd"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.405036 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:00.425451 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:00.428088 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:00.428323 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:00.428598 5321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:00.429060 5321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:00.429159 5321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.429236 5321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:00.429298 5321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.436695 5321 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:45533
I20260928 20:27:00.437279 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:27:00.440596 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5321
I20260928 20:27:00.440714 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:27:00.440920 25323 external_mini_cluster.cc:1428] Setting key 49105c4f1b4d27d7692f02c99197713d
I20260928 20:27:00.448096 5452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:45533 every 8 connection(s)
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
I20260928 20:27:00.469393 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.449885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:43423 (local address 127.24.186.254:33123)
0928 20:27:00.449933 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:00.449935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:00.450801 (+ 866us) server_negotiation.cc:407] Connection header received
0928 20:27:00.451582 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:00.451584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:00.451604 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:00.451637 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:00.452597 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.453566 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.455164 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.455457 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.455509 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.459691 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:00.459694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:00.459695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:00.459705 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:00.462269 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.467741 (+ 5472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.467744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.467745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.467866 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.468314 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.468316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.468316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.468502 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:00.468760 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:00.469175 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 20:27:00.469292 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
I20260928 20:27:00.471957 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.449545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.194:43423)
0928 20:27:00.450638 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:00.450696 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:00.451434 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:00.451897 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:00.451902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:00.452175 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:00.452482 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.452488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.453697 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.453697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.455051 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.455054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.455893 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.455894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.456048 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.456706 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:00.456716 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:00.459543 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:00.462516 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.462530 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.462533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.462989 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.467963 (+ 4974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.467964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.467965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.468184 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.468759 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:00.468770 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:00.468972 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 20:27:00.469106 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 20:27:00.469292 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"thread_start_us":115,"threads_started":1}
I20260928 20:27:00.473325 5453 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:00.473448 5453 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:00.473682 5453 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:00.474174 5102 ts_manager.cc:194] Registered new tserver with Master: 4d51731cb051487cafe8e40afed041bf (127.24.186.194:45533)
I20260928 20:27:00.474684 5102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:43423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:27:00.496202 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:00.628429 5458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:00.628731 5458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:00.628767 5458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:00.631098 5458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:00.631198 5458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:00.631253 5458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:00.634572 5458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:00.634958 5458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:00.635305 5458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:00.652120 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.652437 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.652626 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:00.655733 5458 server_base.cc:1034] running on GCE node
I20260928 20:27:00.656092 5458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:00.656405 5458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:00.663873 5458 hybrid_clock.cc:648] HybridClock initialized: now 1790627220664114 us; error 335 us; skew 500 ppm
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:00.667945 5458 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:00.668493 5458 webserver.cc:460] Webserver started at http://127.24.186.195:45753/ using document root <none> and password file <none>
I20260928 20:27:00.668753 5458 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:00.668802 5458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:00.668927 5458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:00.690939 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "166cf3827a1646918e188d80781b9ea8"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "c45b31974a49724ad7a2a6229d9ec50f"
server_key_iv: "c84f3c4946d5d4f2c824f80484539c9a"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.691531 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "166cf3827a1646918e188d80781b9ea8"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "c45b31974a49724ad7a2a6229d9ec50f"
server_key_iv: "c84f3c4946d5d4f2c824f80484539c9a"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.709848 5458 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.002s sys 0.000s
I20260928 20:27:00.726226 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.726500 5458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 20:27:00.726588 5458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "166cf3827a1646918e188d80781b9ea8"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "c45b31974a49724ad7a2a6229d9ec50f"
server_key_iv: "c84f3c4946d5d4f2c824f80484539c9a"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.726679 5458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:00.742787 5458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:00.745452 5458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:00.745690 5458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:00.745961 5458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:00.746381 5458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:00.746466 5458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.746532 5458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:00.746584 5458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.754051 5458 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45401
I20260928 20:27:00.754573 5458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:27:00.760572 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5458
I20260928 20:27:00.760707 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:27:00.760905 25323 external_mini_cluster.cc:1428] Setting key ee711bbd60635860fd888c08b7b4ef25
I20260928 20:27:00.769556 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45401 every 8 connection(s)
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:00.787771 5318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.770688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:36805 (local address 127.24.186.254:33123)
0928 20:27:00.770735 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:00.770737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:00.772569 (+ 1832us) server_negotiation.cc:407] Connection header received
0928 20:27:00.773295 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:00.773296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:00.773309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:00.773338 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:00.774195 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.774944 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.776568 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.776842 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.776883 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.783608 (+ 6725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:00.783610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:00.783610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:00.783620 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:00.785372 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.785957 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.785959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.785959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.786020 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:00.786320 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:00.786321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:00.786321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:00.786470 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:00.786520 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:00.787557 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 20:27:00.787662 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:27:00.788513 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:00.770359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.195:36805)
0928 20:27:00.771885 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:00.771941 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:00.773174 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:00.773537 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:00.773541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:00.773811 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:00.774090 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.774096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.775063 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.775064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.776463 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:00.776466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:00.776994 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:00.776995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:00.777139 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:00.777770 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:00.777781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:00.783475 (+ 5694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:00.785501 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.785515 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.785519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.785865 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.786084 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:00.786085 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:00.786085 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:00.786216 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:00.786555 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:00.786561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:00.787347 (+ 786us) client_negotiation.cc:769] Sending connection context
0928 20:27:00.787964 (+ 617us) client_negotiation.cc:240] Negotiation successful
0928 20:27:00.788154 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":104,"threads_started":1}
I20260928 20:27:00.789502 5587 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:00.789619 5587 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:00.789819 5587 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:00.790220 5102 ts_manager.cc:194] Registered new tserver with Master: 166cf3827a1646918e188d80781b9ea8 (127.24.186.195:45401)
I20260928 20:27:00.790701 5102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:36805
I20260928 20:27:00.794566 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:27:00.795244 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5184
I20260928 20:27:00.802901 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5321
I20260928 20:27:00.807852 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5458
I20260928 20:27:00.813539 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5070
I20260928 20:27:00.819602 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33123
--webserver_interface=127.24.186.254
--webserver_port=34957
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:33123
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:00.907202 5590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:00.907430 5590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:00.907454 5590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:00.908717 5590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:00.908768 5590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:00.908783 5590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:00.908798 5590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:00.908810 5590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:00.910508 5590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:33123
--ranger_config_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:33123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=34957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:00.910766 5590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:00.911022 5590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:00.916082 5595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:00.916113 5596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:00.916188 5590 server_base.cc:1034] running on GCE node
W20260928 20:27:00.916260 5598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:00.916566 5590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:00.917903 5590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:00.919191 5590 hybrid_clock.cc:648] HybridClock initialized: now 1790627220919151 us; error 87 us; skew 500 ppm
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:00.921950 5590 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:00.922489 5590 webserver.cc:460] Webserver started at http://127.24.186.254:34957/ using document root <none> and password file <none>
I20260928 20:27:00.922752 5590 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:00.922832 5590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:27:00 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627220, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:00.977972 5590 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:00.978929 5606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:00.979140 5590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:00.979250 5590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "cae45b5f68254d98a07397a08d658bd3"
format_stamp: "Formatted at 2026-09-28 20:26:59 on dist-test-slave-k477"
server_key: "d9dd9188ffb72c591cc09df650f88e68"
server_key_iv: "08ab1b7212ab47eee4d8aef7c3df8582"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:00.979650 5590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:00.997617 5590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:01.000654 5590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:01.000800 5590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:01.007084 5590 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:33123
I20260928 20:27:01.007206 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:33123 every 8 connection(s)
I20260928 20:27:01.007579 5590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:01.009006 5659 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:27:01.009851 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Bootstrap starting.
I20260928 20:27:01.016691 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5590
I20260928 20:27:01.017071 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:36545
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:33123
--webserver_port=35175
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:01.020295 5659 log.cc:824] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:01.022671 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:01.023041 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Bootstrap complete.
I20260928 20:27:01.027426 5659 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:27:01.027894 5659 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae45b5f68254d98a07397a08d658bd3, State: Initialized, Role: FOLLOWER
I20260928 20:27:01.028097 5659 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:27:01.028188 5659 raft_consensus.cc:388] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:01.028246 5659 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:01.028308 5659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:27:01.029893 5659 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:27:01.030010 5659 leader_election.cc:302] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cae45b5f68254d98a07397a08d658bd3; no voters:
I20260928 20:27:01.030189 5659 leader_election.cc:288] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:27:01.030278 5663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:27:01.030429 5659 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:01.030514 5663 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [term 2 LEADER]: Becoming Leader. State: Replica: cae45b5f68254d98a07397a08d658bd3, State: Running, Role: LEADER
I20260928 20:27:01.030625 5663 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } }
I20260928 20:27:01.030968 5664 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cae45b5f68254d98a07397a08d658bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } } }
I20260928 20:27:01.031052 5664 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:01.031301 5665 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae45b5f68254d98a07397a08d658bd3. Latest consensus state: current_term: 2 leader_uuid: "cae45b5f68254d98a07397a08d658bd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cae45b5f68254d98a07397a08d658bd3" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 33123 } } }
I20260928 20:27:01.031414 5665 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3 [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:01.031665 5670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:01.032191 5670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:01.032370 5670 catalog_manager.cc:1259] Loaded cluster ID: 2725db6bc60b4db1be97ab0a5958a6e9
I20260928 20:27:01.032446 5670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:27:01.032549 5659 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 20:27:01.036540 5670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:01.037222 5670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Loaded TSK: 0
I20260928 20:27:01.037528 5670 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:27:01.117357 5661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:01.117717 5661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:01.117769 5661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:01.119134 5661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:01.119217 5661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:01.119266 5661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:01.122483 5661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:36545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=35175
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:01.122831 5661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:01.123092 5661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:01.128750 5699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:01.128866 5697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:01.128757 5696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:01.129344 5661 server_base.cc:1034] running on GCE node
I20260928 20:27:01.129601 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:01.129884 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:01.131068 5661 hybrid_clock.cc:648] HybridClock initialized: now 1790627221131049 us; error 24 us; skew 500 ppm
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:01.134106 5661 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:01.134653 5661 webserver.cc:460] Webserver started at http://127.24.186.193:35175/ using document root <none> and password file <none>
I20260928 20:27:01.134869 5661 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:01.134917 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:01.163689 5661 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 20:27:01.164885 5707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.165108 5661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:27:01.165195 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b73457a4be4c4a809ff80609effcd61c"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "e4a0cb6032ddc0405223caac9af24e3b"
server_key_iv: "1cd903e98021756dd4b15535a2462365"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:01.165674 5661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:01.176900 5661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:01.179862 5661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:01.180150 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:01.180512 5661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:01.181088 5661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:01.181165 5661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.181237 5661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:01.181260 5661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.190579 5661 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:36545
I20260928 20:27:01.191102 5661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:01.195768 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5661
I20260928 20:27:01.196161 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:45533
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:33123
--webserver_port=37459
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:01.197474 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:36545 every 8 connection(s)
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:01.212803 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.196791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.193:48303)
0928 20:27:01.198631 (+ 1840us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:01.198686 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:01.199336 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:01.200182 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:01.200186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:01.200452 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:01.200767 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.200773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.203353 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.203354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.204628 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.204632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.205638 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.205639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.205767 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.206408 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:01.206427 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:01.208316 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:01.210635 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.210642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.210644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.211110 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.211420 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.211421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.211422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.211557 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.212119 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:01.212125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:01.212276 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 20:27:01.212387 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:27:01.212561 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1785,"thread_start_us":117,"threads_started":1}
I20260928 20:27:01.214756 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.197204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:48303 (local address 127.24.186.254:33123)
0928 20:27:01.199581 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:01.199590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:01.199628 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:27:01.199781 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:01.199787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:01.199871 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:01.200011 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:01.200920 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.203201 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.204745 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.205406 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.205515 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.208469 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:01.208474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:01.208475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:01.208490 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:01.210525 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.211218 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.211221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.211222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.211339 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.211635 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.211636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.211637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.211989 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:01.212087 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:01.213451 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0928 20:27:01.214274 (+ 823us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2318,"thread_start_us":114,"threads_started":1}
I20260928 20:27:01.220287 5821 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:01.220445 5821 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.220675 5821 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.221668 5821 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:27:01.221771 5821 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.221956 5821 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:01.222385 5821 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.222476 5821 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.222800 5821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:27:01.334239 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:01.334553 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:01.334609 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:01.337270 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:01.337423 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:01.337491 5825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:01.341591 5825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:45533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=37459
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:01.341962 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:01.342264 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:01.348862 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:01.349051 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:01.349481 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:01.349607 5825 server_base.cc:1034] running on GCE node
I20260928 20:27:01.349817 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:01.350078 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:01.351248 5825 hybrid_clock.cc:648] HybridClock initialized: now 1790627221351195 us; error 65 us; skew 500 ppm
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:01.354692 5825 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:01.355358 5825 webserver.cc:460] Webserver started at http://127.24.186.194:37459/ using document root <none> and password file <none>
I20260928 20:27:01.355654 5825 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:01.355755 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:01.371568 5825 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 20:27:01.372643 5841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.372833 5825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:27:01.372926 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "4d51731cb051487cafe8e40afed041bf"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "633a766531670dfd430528e3bbbd5b17"
server_key_iv: "ce59273865f7c3224f5f3fcd982edafd"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:01.373328 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:01.429131 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:01.431486 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:01.431689 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:01.432080 5825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:01.432473 5825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:01.432538 5825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.432578 5825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:01.432605 5825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.440572 5825 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:45533
I20260928 20:27:01.441001 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:27:01.441864 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:45533 every 8 connection(s)
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:01.450968 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5825
I20260928 20:27:01.451270 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.24.186.212:47063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:45401
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:33123
--webserver_port=45753
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:01.454028 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.442273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:37357 (local address 127.24.186.254:33123)
0928 20:27:01.442317 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:01.442320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:01.442330 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:01.442844 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:01.442846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:01.443028 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:01.443176 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:01.444242 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.444892 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.446027 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.446258 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.446337 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.448879 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:01.448881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:01.448881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:01.448890 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:01.450349 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.452741 (+ 2392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.452743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.452743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.452820 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.453176 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.453177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.453177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.453330 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:01.453391 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:01.453818 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 20:27:01.453934 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:27:01.454219 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.441943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.194:37357)
0928 20:27:01.442092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:01.442134 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:01.442726 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:01.443312 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:01.443318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:01.443604 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:01.444097 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.444108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.444998 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.444999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.445923 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.445926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.446379 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.446381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.446502 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.447133 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:01.447145 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:01.448764 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:01.450465 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.450472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.450475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.450725 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.452915 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.452917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.452917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.453094 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.453442 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:01.453448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:01.453647 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 20:27:01.453795 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 20:27:01.453961 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260928 20:27:01.454939 5955 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:01.455061 5955 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.455281 5955 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.456964 5955 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:27:01.457062 5955 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.457139 5955 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:01.457664 5955 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.457808 5955 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.458221 5955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:27:01.560101 5958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:01.560420 5958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:01.560672 5958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:01.562216 5958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:01.562273 5958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:01.562314 5958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:01.564576 5958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.24.186.212:47063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:45401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=45753
--tserver_master_addrs=127.24.186.254:33123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:01.564872 5958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:01.565166 5958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:01.570376 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:01.570598 5963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:01.570971 5958 server_base.cc:1034] running on GCE node
W20260928 20:27:01.571218 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:01.571480 5958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:01.571766 5958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:01.572969 5958 hybrid_clock.cc:648] HybridClock initialized: now 1790627221572955 us; error 28 us; skew 500 ppm
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:01.575603 5958 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:01.576155 5958 webserver.cc:460] Webserver started at http://127.24.186.195:45753/ using document root <none> and password file <none>
I20260928 20:27:01.576416 5958 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:01.576495 5958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:01.593268 5958 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:01.597453 5975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.597709 5958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260928 20:27:01.597795 5958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "166cf3827a1646918e188d80781b9ea8"
format_stamp: "Formatted at 2026-09-28 20:27:00 on dist-test-slave-k477"
server_key: "c45b31974a49724ad7a2a6229d9ec50f"
server_key_iv: "c84f3c4946d5d4f2c824f80484539c9a"
server_key_version: "kuduclusterkey@0"
I20260928 20:27:01.598282 5958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:01.619065 5958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:01.621941 5958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:01.622171 5958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:01.622468 5958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:01.622874 5958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:01.622954 5958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.623016 5958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:01.623060 5958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:01.630514 5958 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45401
I20260928 20:27:01.631153 5958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:27:01.631302 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 5958
I20260928 20:27:01.637295 6088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45401 every 8 connection(s)
I20260928 20:27:01.642254 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.632335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37486 (local address 127.24.186.254:33123)
0928 20:27:01.632384 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:01.632386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:01.634406 (+ 2020us) server_negotiation.cc:407] Connection header received
0928 20:27:01.634491 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:01.634493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:01.634514 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:01.634581 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:01.635260 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.636099 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.637194 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.637468 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.637554 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.639286 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:01.639287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:01.639287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:01.639295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:01.641024 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.641408 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.641409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.641409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.641468 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.641697 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.641698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.641698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.641843 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:01.642014 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:01.642084 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 20:27:01.642161 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 20:27:01 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627221, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:01.648269 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.635914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:36855 (local address 127.24.186.254:33123)
0928 20:27:01.637752 (+ 1838us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:01.637754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:01.637761 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:27:01.637781 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:01.637782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:01.637795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:01.637864 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:01.638544 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.639038 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.640547 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.640862 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.640935 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.644435 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:01.644436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:01.644437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:01.644444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:01.646230 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.647003 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.647004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.647004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.647070 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:01.647359 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:01.647360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:01.647360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:01.647485 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:01.647560 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:01.648072 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 20:27:01.648164 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":55,"threads_started":1}
I20260928 20:27:01.648977 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:01.636137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:33123 (local address 127.24.186.195:36855)
0928 20:27:01.636316 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:01.636373 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:01.637319 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:01.637985 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:01.637990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:01.638240 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:01.638441 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.638445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.639141 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.639142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.640447 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:01.640451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:01.641005 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:01.641007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:01.641729 (+ 722us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:01.642717 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:01.642727 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:01.644343 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:01.646338 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.646345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.646349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.646906 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.647154 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:01.647154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:01.647155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:01.647292 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:01.647587 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:01.647593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:01.647758 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 20:27:01.647929 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 20:27:01.648739 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 20:27:01.650025 6089 heartbeater.cc:347] Connected to a master server at 127.24.186.254:33123
I20260928 20:27:01.650142 6089 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.650328 6089 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.650979 6089 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:33123 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 20:27:01.651065 6089 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.651134 6089 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:01.651435 6089 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:01.651527 6089 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
W20260928 20:27:01.651837 6089 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:27:02.223008 5821 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:02.223178 5821 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:02.223641 5623 ts_manager.cc:194] Registered new tserver with Master: b73457a4be4c4a809ff80609effcd61c (127.24.186.193:36545)
I20260928 20:27:02.224685 5623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:48303
Sep 28 20:27:02 dist-test-slave-k477 krb5kdc[4626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790627222, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:02.458874 5955 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:02.459038 5955 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:02.459349 5622 ts_manager.cc:194] Registered new tserver with Master: 4d51731cb051487cafe8e40afed041bf (127.24.186.194:45533)
I20260928 20:27:02.459954 5622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:37357
I20260928 20:27:02.652215 6089 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:02.652839 6089 heartbeater.cc:511] Master 127.24.186.254:33123 requested a full tablet report, sending...
I20260928 20:27:02.653949 5622 ts_manager.cc:194] Registered new tserver with Master: 166cf3827a1646918e188d80781b9ea8 (127.24.186.195:45401)
I20260928 20:27:02.654834 5622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:36855
I20260928 20:27:02.659508 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 20:27:02 dist-test-slave-k477 krb5kdc[4626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627222, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for HTTP/127.24.186.212@KRBTEST.COM
I20260928 20:27:02.677261 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.660906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.24.186.254:33123)
0928 20:27:02.661538 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.661541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.661749 (+ 208us) server_negotiation.cc:407] Connection header received
0928 20:27:02.661777 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.661778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.661790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.661869 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:02.662614 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.663397 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.664483 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.664793 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.664851 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.673448 (+ 8597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:02.673451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:02.673451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:02.673462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:02.675382 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:02.675912 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:02.675913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:02.675914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:02.675982 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:02.676173 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:02.676174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:02.676174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:02.676318 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:02.676406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.676984 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.677135 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":69,"threads_started":1}
I20260928 20:27:02.680588 5622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:27:02.683044 5622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:27:02.698066 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:45533 (local address 127.0.0.1:58286)
0928 20:27:02.693434 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.693443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:02.693483 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:02.693767 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:02.693770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:02.693773 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:02.693852 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.693854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.695282 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.695283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.696666 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.696669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.697730 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.697731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.697829 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.697844 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:27:02.697913 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:27:02.697969 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":90,"threads_started":1}
I20260928 20:27:02.698326 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.24.186.194:45533)
0928 20:27:02.693593 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.693597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.693605 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:02.693635 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.693637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.693649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.693745 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:02.693943 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.695197 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.696766 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.697561 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.697636 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.697694 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.698203 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.698252 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":903,"thread_start_us":51,"threads_started":1}
I20260928 20:27:02.699071 5889 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1d87a79eb444799fee522d7770906 (DEFAULT_TABLE table=test-table [id=c38c9c9439d64c0cafbc416301bad3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:02.699350 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36545 (local address 127.0.0.1:54070)
0928 20:27:02.693200 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.693243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:02.693308 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:02.694228 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:02.694229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:02.694231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:02.694277 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.694279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.695870 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.695871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.696948 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.696951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.699116 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.699118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.699186 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.699190 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:02.699232 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 20:27:02.699273 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":106,"threads_started":1}
I20260928 20:27:02.699465 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1d87a79eb444799fee522d7770906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:02.699756 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.24.186.193:36545)
0928 20:27:02.694037 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.694042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.694051 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:27:02.694096 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.694098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.694113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.694356 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:02.694436 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.695732 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.697051 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.699007 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.699536 (+ 529us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.699613 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.699642 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.699682 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":55,"threads_started":1}
I20260928 20:27:02.700387 5755 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1d87a79eb444799fee522d7770906 (DEFAULT_TABLE table=test-table [id=c38c9c9439d64c0cafbc416301bad3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:02.700716 5755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1d87a79eb444799fee522d7770906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:02.704507 6123 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Bootstrap starting.
I20260928 20:27:02.704748 6122 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: Bootstrap starting.
I20260928 20:27:02.705163 6123 tablet_bootstrap.cc:654] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:02.705384 6122 tablet_bootstrap.cc:654] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:02.705480 6123 log.cc:824] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:02.705713 6122 log.cc:824] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:02.706503 6122 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: No bootstrap required, opened a new log
I20260928 20:27:02.706580 6122 ts_tablet_manager.cc:1397] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:27:02.708813 6122 raft_consensus.cc:348] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.709018 6122 raft_consensus.cc:374] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:02.709050 6122 raft_consensus.cc:729] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d51731cb051487cafe8e40afed041bf, State: Initialized, Role: FOLLOWER
I20260928 20:27:02.709622 6122 consensus_queue.cc:260] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.709939 6122 ts_tablet_manager.cc:1428] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:27:02.710075 5955 heartbeater.cc:503] Master 127.24.186.254:33123 was elected leader, sending a full tablet report...
I20260928 20:27:02.711663 6123 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: No bootstrap required, opened a new log
I20260928 20:27:02.711745 6123 ts_tablet_manager.cc:1397] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 20:27:02.711941 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45401 (local address 127.0.0.1:46658)
0928 20:27:02.693871 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.693881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:02.693929 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:02.702992 (+ 9063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:02.702994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:02.702997 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:02.703048 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.703050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.708502 (+ 5452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.708504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.709719 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.709722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.711502 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.711504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.711716 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.711724 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 20:27:02.711775 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:27:02.711858 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":90,"threads_started":1}
I20260928 20:27:02.712596 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.692971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.24.186.195:45401)
0928 20:27:02.702774 (+ 9803us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.702779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.702792 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:27:02.702830 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.702841 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.702860 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.703126 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:02.703215 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.708368 (+ 5153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.710054 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.711377 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.712340 (+ 963us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.712420 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.712451 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.712502 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9762,"thread_start_us":68,"threads_started":1}
I20260928 20:27:02.714756 6023 tablet_service.cc:1467] Processing CreateTablet for tablet 78a1d87a79eb444799fee522d7770906 (DEFAULT_TABLE table=test-table [id=c38c9c9439d64c0cafbc416301bad3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:02.714931 6123 raft_consensus.cc:348] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.715130 6123 raft_consensus.cc:374] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:02.715163 6023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78a1d87a79eb444799fee522d7770906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:02.715169 6123 raft_consensus.cc:729] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73457a4be4c4a809ff80609effcd61c, State: Initialized, Role: FOLLOWER
I20260928 20:27:02.715287 6123 consensus_queue.cc:260] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.715585 6123 ts_tablet_manager.cc:1428] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:27:02.715718 5821 heartbeater.cc:503] Master 127.24.186.254:33123 was elected leader, sending a full tablet report...
I20260928 20:27:02.720793 6128 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Bootstrap starting.
I20260928 20:27:02.721542 6128 tablet_bootstrap.cc:654] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:02.721899 6128 log.cc:824] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:02.723989 6128 tablet_bootstrap.cc:492] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: No bootstrap required, opened a new log
I20260928 20:27:02.724066 6128 ts_tablet_manager.cc:1397] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:27:02.727368 6128 raft_consensus.cc:348] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.727607 6128 raft_consensus.cc:374] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:02.727658 6128 raft_consensus.cc:729] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166cf3827a1646918e188d80781b9ea8, State: Initialized, Role: FOLLOWER
I20260928 20:27:02.727837 6128 consensus_queue.cc:260] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.728163 6128 ts_tablet_manager.cc:1428] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:27:02.728292 6089 heartbeater.cc:503] Master 127.24.186.254:33123 was elected leader, sending a full tablet report...
I20260928 20:27:02.809762 6126 raft_consensus.cc:482] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:02.809926 6126 raft_consensus.cc:504] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.810333 6126 leader_election.cc:288] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b73457a4be4c4a809ff80609effcd61c (127.24.186.193:36545), 166cf3827a1646918e188d80781b9ea8 (127.24.186.195:45401)
I20260928 20:27:02.815923 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.810966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:38479 (local address 127.24.186.195:45401)
0928 20:27:02.811011 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.811013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.811391 (+ 378us) server_negotiation.cc:407] Connection header received
0928 20:27:02.811496 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.811497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.811506 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.811535 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:02.812056 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.812817 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.814895 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.815482 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.815534 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.815542 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.815773 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.815843 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:27:02.816409 6131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.810638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45401 (local address 127.24.186.194:38479)
0928 20:27:02.811325 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.811337 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:02.811420 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:02.811855 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:02.811857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:02.811859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:02.811942 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.811945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.812928 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.812929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.814655 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.814657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.815621 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.815622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.815695 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.815698 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:27:02.816260 (+ 562us) client_negotiation.cc:240] Negotiation successful
0928 20:27:02.816310 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":115,"threads_started":1}
I20260928 20:27:02.816757 6043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1d87a79eb444799fee522d7770906" candidate_uuid: "4d51731cb051487cafe8e40afed041bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166cf3827a1646918e188d80781b9ea8" is_pre_election: true
I20260928 20:27:02.816941 6043 raft_consensus.cc:2454] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d51731cb051487cafe8e40afed041bf in term 0.
I20260928 20:27:02.817164 5844 leader_election.cc:302] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166cf3827a1646918e188d80781b9ea8, 4d51731cb051487cafe8e40afed041bf; no voters:
I20260928 20:27:02.817302 6126 raft_consensus.cc:2781] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:02.817345 6126 raft_consensus.cc:482] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:02.817377 6126 raft_consensus.cc:3037] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:02.817880 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.811324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:34255 (local address 127.24.186.193:36545)
0928 20:27:02.811384 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.811386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.814268 (+ 2882us) server_negotiation.cc:407] Connection header received
0928 20:27:02.814470 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.814471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.814483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.814537 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:02.814873 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.815603 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.816918 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.817505 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.817577 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.817586 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.817761 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.817800 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:27:02.818172 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.811140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36545 (local address 127.24.186.194:34255)
0928 20:27:02.814203 (+ 3063us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.814215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:02.814270 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:02.814731 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:02.814732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:02.814734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:02.814801 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.814802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.815703 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.815704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.816732 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:02.816734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.817617 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.817618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:02.817682 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.817685 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:27:02.818040 (+ 355us) client_negotiation.cc:240] Negotiation successful
0928 20:27:02.818096 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3041,"thread_start_us":66,"threads_started":1}
I20260928 20:27:02.818459 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1d87a79eb444799fee522d7770906" candidate_uuid: "4d51731cb051487cafe8e40afed041bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73457a4be4c4a809ff80609effcd61c" is_pre_election: true
I20260928 20:27:02.818602 5769 raft_consensus.cc:2454] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d51731cb051487cafe8e40afed041bf in term 0.
I20260928 20:27:02.818732 6126 raft_consensus.cc:504] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.818858 6126 leader_election.cc:288] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [CANDIDATE]: Term 1 election: Requested vote from peers b73457a4be4c4a809ff80609effcd61c (127.24.186.193:36545), 166cf3827a1646918e188d80781b9ea8 (127.24.186.195:45401)
I20260928 20:27:02.819031 5769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1d87a79eb444799fee522d7770906" candidate_uuid: "4d51731cb051487cafe8e40afed041bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73457a4be4c4a809ff80609effcd61c"
I20260928 20:27:02.819093 5769 raft_consensus.cc:3037] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:02.819048 6043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78a1d87a79eb444799fee522d7770906" candidate_uuid: "4d51731cb051487cafe8e40afed041bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166cf3827a1646918e188d80781b9ea8"
I20260928 20:27:02.819150 6043 raft_consensus.cc:3037] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:02.819768 5769 raft_consensus.cc:2454] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d51731cb051487cafe8e40afed041bf in term 1.
I20260928 20:27:02.819984 5845 leader_election.cc:302] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d51731cb051487cafe8e40afed041bf, b73457a4be4c4a809ff80609effcd61c; no voters:
I20260928 20:27:02.820073 6043 raft_consensus.cc:2454] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d51731cb051487cafe8e40afed041bf in term 1.
I20260928 20:27:02.820111 6126 raft_consensus.cc:2781] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:02.820258 6126 raft_consensus.cc:686] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [term 1 LEADER]: Becoming Leader. State: Replica: 4d51731cb051487cafe8e40afed041bf, State: Running, Role: LEADER
I20260928 20:27:02.820343 6126 consensus_queue.cc:237] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } }
I20260928 20:27:02.821161 5622 catalog_manager.cc:5495] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d51731cb051487cafe8e40afed041bf (127.24.186.194). New cstate: current_term: 1 leader_uuid: "4d51731cb051487cafe8e40afed041bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d51731cb051487cafe8e40afed041bf" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 45533 } health_report { overall_health: HEALTHY } } }
I20260928 20:27:02.849391 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.845801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.24.186.194:45533)
0928 20:27:02.845880 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:02.845883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:02.845893 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:02.845945 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:02.845947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:02.845970 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:02.846058 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:02.846224 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.847233 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.848410 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:02.848715 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:02.848795 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:02.849000 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:02.849142 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:02.849229 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:02.849280 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 20:27:02.849304 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:27:02.851485 6043 raft_consensus.cc:1261] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 1 FOLLOWER]: Refusing update from remote peer 4d51731cb051487cafe8e40afed041bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:02.851899 6126 consensus_queue.cc:1035] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "166cf3827a1646918e188d80781b9ea8" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:02.852133 5769 raft_consensus.cc:1261] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 1 FOLLOWER]: Refusing update from remote peer 4d51731cb051487cafe8e40afed041bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:02.852530 6126 consensus_queue.cc:1035] T 78a1d87a79eb444799fee522d7770906 P 4d51731cb051487cafe8e40afed041bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73457a4be4c4a809ff80609effcd61c" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:02.855121 6136 mvcc.cc:205] Tried to move back new op lower bound from 7334409104796872704 to 7334409104672628736. Current Snapshot: MvccSnapshot[applied={T|T < 7334409104796872704}]
I20260928 20:27:02.855698 6137 mvcc.cc:205] Tried to move back new op lower bound from 7334409104796872704 to 7334409104672628736. Current Snapshot: MvccSnapshot[applied={T|T < 7334409104796872704}]
I20260928 20:27:02.857342 6139 mvcc.cc:205] Tried to move back new op lower bound from 7334409104796872704 to 7334409104672628736. Current Snapshot: MvccSnapshot[applied={T|T < 7334409104796872704}]
I20260928 20:27:02.859248 5622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:02.859346 5622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:02.860129 5622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cae45b5f68254d98a07397a08d658bd3: Sending DeleteTablet for 3 replicas of tablet 78a1d87a79eb444799fee522d7770906
I20260928 20:27:02.860684 5755 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a1d87a79eb444799fee522d7770906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:02 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:54070
I20260928 20:27:02.860864 6023 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a1d87a79eb444799fee522d7770906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:02 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:46658
I20260928 20:27:02.860956 6146 tablet_replica.cc:330] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: stopping tablet replica
I20260928 20:27:02.861050 6146 raft_consensus.cc:2229] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:02.861096 5889 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a1d87a79eb444799fee522d7770906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:02 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:58286
I20260928 20:27:02.861160 6146 raft_consensus.cc:2258] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:02.861618 6146 ts_tablet_manager.cc:1905] T 78a1d87a79eb444799fee522d7770906 P b73457a4be4c4a809ff80609effcd61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:02.861851 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5661
W20260928 20:27:02.867914 5610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:36545 (error 108)
I20260928 20:27:02.868350 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5825
I20260928 20:27:02.868935 6147 tablet_replica.cc:330] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: stopping tablet replica
I20260928 20:27:02.869076 6147 raft_consensus.cc:2229] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:02.869192 6147 raft_consensus.cc:2258] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:02.869709 6147 ts_tablet_manager.cc:1905] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:02.874073 6147 ts_tablet_manager.cc:1918] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:02.874171 6147 log.cc:1196] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/78a1d87a79eb444799fee522d7770906
I20260928 20:27:02.874464 6147 ts_tablet_manager.cc:1939] T 78a1d87a79eb444799fee522d7770906 P 166cf3827a1646918e188d80781b9ea8: Deleting consensus metadata
I20260928 20:27:02.876114 5609 catalog_manager.cc:4841] TS 166cf3827a1646918e188d80781b9ea8 (127.24.186.195:45401): tablet 78a1d87a79eb444799fee522d7770906 (table test-table [id=c38c9c9439d64c0cafbc416301bad3b1]) successfully deleted
I20260928 20:27:02.877395 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.877111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36545 (local address 127.0.0.1:54072)
0928 20:27:02.877213 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.877296 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:36545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 20:27:02.877400 6117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:02.877233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:45533 (local address 127.0.0.1:58300)
0928 20:27:02.877287 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:02.877322 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.194:45533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 20:27:02.877628 5607 catalog_manager.cc:4584] TS 4d51731cb051487cafe8e40afed041bf (127.24.186.194:45533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78a1d87a79eb444799fee522d7770906: Network error: Client connection negotiation failed: client connection to 127.24.186.194:45533: connect: Connection refused (error 111)
I20260928 20:27:02.878057 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5958
I20260928 20:27:02.883859 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 5590
I20260928 20:27:02.890406 25323 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 20:27:03.235698 25323 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 20:27:03.235857 25323 mini_ranger.cc:66] Stopping Ranger...
I20260928 20:27:03.581292 25323 mini_ranger.cc:68] Stopped Ranger
2026-09-28 20:27:03.581 UTC [4656] LOG: received smart shutdown request
2026-09-28 20:27:03.594 UTC [4656] LOG: background worker "logical replication launcher" (PID 4665) exited with exit code 1
2026-09-28 20:27:03.594 UTC [4660] LOG: shutting down
2026-09-28 20:27:03.874 UTC [4656] LOG: database system is shut down
2026-09-28T20:27:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51462 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:27:04 dist-test-slave-k477 krb5kdc[6158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:27:04 dist-test-slave-k477 krb5kdc[6158](info): set up 2 sockets
Sep 28 20:27:04 dist-test-slave-k477 krb5kdc[6158](info): commencing operation
krb5kdc: starting...
W20260928 20:27:05.072367 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:27:05.207427 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:45063
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:05.294504 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:05.294776 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:05.294816 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:05.296094 6171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:05.296180 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:05.296204 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:05.296231 6171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:05.296263 6171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:05.297863 6171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:05.298162 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:05.298404 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:05.303831 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:05.303786 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:05.303789 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:05.304085 6171 server_base.cc:1034] running on GCE node
I20260928 20:27:05.304340 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:05.304649 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:05.305831 6171 hybrid_clock.cc:648] HybridClock initialized: now 1790627225305808 us; error 39 us; skew 500 ppm
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:05.308642 6171 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:05.309227 6171 webserver.cc:460] Webserver started at http://127.24.186.254:39921/ using document root <none> and password file <none>
I20260928 20:27:05.309506 6171 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:05.309582 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:05.309746 6171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:05.310920 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "383277af132c4187aba71dc3f8a25c44"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "07f9f896e19170aa37ba6d0c1186725a"
server_key_iv: "529639a9d1061bddc88b519f6196f2ec"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.311313 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "383277af132c4187aba71dc3f8a25c44"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "07f9f896e19170aa37ba6d0c1186725a"
server_key_iv: "529639a9d1061bddc88b519f6196f2ec"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.312829 6171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:27:05.313694 6186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:05.313908 6171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:05.314018 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "383277af132c4187aba71dc3f8a25c44"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "07f9f896e19170aa37ba6d0c1186725a"
server_key_iv: "529639a9d1061bddc88b519f6196f2ec"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.314105 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:05.346485 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:05.348963 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:05.349145 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:05.353364 6171 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:45063
I20260928 20:27:05.353407 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:45063 every 8 connection(s)
I20260928 20:27:05.353787 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:05.354668 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:05.356387 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:27:05.363457 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6171
I20260928 20:27:05.363651 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:05.363977 25323 external_mini_cluster.cc:1428] Setting key 2dd3d2bccbbb5a801d9047263bac5870
I20260928 20:27:05.369724 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.357247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.24.186.254:45063)
0928 20:27:05.357414 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.357422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.357557 (+ 135us) server_negotiation.cc:407] Connection header received
0928 20:27:05.358076 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.358078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.358098 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.358193 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.359154 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.360363 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.361307 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.361635 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.361722 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.364430 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.364437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.364437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.364445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.367747 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.368369 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.368372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.368372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.368465 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.368819 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.368820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.368820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.369059 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.369128 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.369389 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.369546 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260928 20:27:05.369830 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.357153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:45063 (local address 127.0.0.1:60164)
0928 20:27:05.357369 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.357416 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.357899 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.358334 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.358339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.358746 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.358980 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.358985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.360480 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.360481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.361218 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.361220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.361729 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.361730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.361794 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.362386 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.362395 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.364321 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.367931 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.367939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.367942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.368254 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.368543 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.368544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.368544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.368732 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.369142 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.369150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.369240 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.369367 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.369635 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":75,"thread_start_us":117,"threads_started":1}
I20260928 20:27:05.370675 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:27:05.371102 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.370936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35564)
0928 20:27:05.371013 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.371070 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 20:27:05.371276 6187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.186.253@KRBTEST.COM; defaulting to no policy
I20260928 20:27:05.372668 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.372496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35566)
0928 20:27:05.372581 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.372627 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 20:27:05.372898 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:34059: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
I20260928 20:27:05.391680 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:34059
--webserver_interface=127.24.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:05.422200 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } attempt: 1
I20260928 20:27:05.422885 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.422719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35572)
0928 20:27:05.422788 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.422845 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:27:05.423337 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.423241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35586)
0928 20:27:05.423286 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.423316 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:27:05.423499 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:34059: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
W20260928 20:27:05.486334 6246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:05.486598 6246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:05.486627 6246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:05.488368 6246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:05.488426 6246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:05.488441 6246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:05.488456 6246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:05.488467 6246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:05.490548 6246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:05.490852 6246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:05.491137 6246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:27:05.491895 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } attempt: 2
I20260928 20:27:05.492429 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.492241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35594)
0928 20:27:05.492321 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.492372 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 20:27:05.492795 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.492691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35608)
0928 20:27:05.492743 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.492775 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 20:27:05.492897 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:34059: Network error: Client connection negotiation failed: client connection to 127.24.186.253:34059: connect: Connection refused (error 111)
W20260928 20:27:05.496346 6254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:05.496461 6246 server_base.cc:1034] running on GCE node
W20260928 20:27:05.496382 6251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:05.496587 6252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:05.496861 6246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:05.497187 6246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:05.498387 6246 hybrid_clock.cc:648] HybridClock initialized: now 1790627225498436 us; error 113 us; skew 500 ppm
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:05.501605 6246 init.cc:377] Logged in from keytab as kudu/127.24.186.253@KRBTEST.COM (short username kudu)
I20260928 20:27:05.502132 6246 webserver.cc:460] Webserver started at http://127.24.186.253:36017/ using document root <none> and password file <none>
I20260928 20:27:05.502383 6246 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:05.502453 6246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:05.502609 6246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:05.503727 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data/instance:
uuid: "d81dd0f4ec2b4d548b8aed49d251b456"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "8c226d53ea58a417ffd8f87715fe931e"
server_key_iv: "88bfe537518089d7870092fcb83c588b"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.504176 6246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal/instance:
uuid: "d81dd0f4ec2b4d548b8aed49d251b456"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "8c226d53ea58a417ffd8f87715fe931e"
server_key_iv: "88bfe537518089d7870092fcb83c588b"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.505834 6246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:05.506958 6261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:05.507248 6246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:05.507359 6246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "d81dd0f4ec2b4d548b8aed49d251b456"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "8c226d53ea58a417ffd8f87715fe931e"
server_key_iv: "88bfe537518089d7870092fcb83c588b"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.507472 6246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:05.525270 6246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:05.528847 6246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:05.529099 6246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:05.533375 6246 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.253:34059
I20260928 20:27:05.533452 6313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.253:34059 every 8 connection(s)
I20260928 20:27:05.533849 6246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
I20260928 20:27:05.534821 6314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:05.536502 6314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:27:05.537216 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6246
I20260928 20:27:05.537357 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-1/wal/instance
I20260928 20:27:05.537550 25323 external_mini_cluster.cc:1428] Setting key a6084779c0728e3dd5f2d25d3fd4b934
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.252@KRBTEST.COM; defaulting to no policy
I20260928 20:27:05.547672 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.537306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:45063 (local address 127.0.0.1:60174)
0928 20:27:05.537576 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.537608 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.538113 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.538369 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.538373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.538654 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.538851 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.538855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.539562 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.539563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.540419 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.540422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.540937 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.540938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.541061 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.541642 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.541651 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.543147 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.545178 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.545186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.545189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.545985 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.546278 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.546279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.546279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.546477 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.547103 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.547109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.547246 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.547344 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.547492 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260928 20:27:05.547850 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.537342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.24.186.254:45063)
0928 20:27:05.537418 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.537420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.537698 (+ 278us) server_negotiation.cc:407] Connection header received
0928 20:27:05.538233 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.538235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.538248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.538289 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.538949 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.539450 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.540531 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.540841 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.540913 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.543249 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.543252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.543252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.543261 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.545054 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.546093 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.546095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.546096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.546179 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.546734 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.546738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.546738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.547012 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.547097 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.547398 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.547486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:27:05.548570 6314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.252@KRBTEST.COM" created.
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:27:05.557637 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.548967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35620)
0928 20:27:05.549036 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.549047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.549092 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.549384 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.549384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.549555 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.549604 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.549606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.550682 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.550683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.551188 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.551190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.551688 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.551689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.551760 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.552297 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.552303 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.553531 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.556195 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.556197 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.556197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.556391 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.556830 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.556831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.556831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.556895 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.557329 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.557330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.557374 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.557455 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.557559 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 20:27:05.557799 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.549071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.24.186.253:34059)
0928 20:27:05.549242 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.549246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.549263 (+ 17us) server_negotiation.cc:407] Connection header received
0928 20:27:05.549281 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.549286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.549297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.549384 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.549708 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.550567 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.551272 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.551598 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.551690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.553719 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.553730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.553730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.553739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.556043 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.556484 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.556488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.556489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.556724 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.556973 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.556974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.556975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.557215 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.557315 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.557511 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.557637 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260928 20:27:05.558730 6314 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:27:05.559176 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.559023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41496)
0928 20:27:05.559085 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.559141 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 20:27:05.559340 6262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
I20260928 20:27:05.559855 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.559698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41500)
0928 20:27:05.559756 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.559787 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 20:27:05.559978 6314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
I20260928 20:27:05.572270 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:38999
--webserver_interface=127.24.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:05.584064 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } attempt: 3
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:27:05.591941 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.584659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35622)
0928 20:27:05.584722 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.584736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.584802 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.584999 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.585000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.585171 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.585240 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.585242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.585998 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.585999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.586512 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.586514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.586921 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.586922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.587004 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.587485 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.587490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.588987 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.590803 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.590804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.590805 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.590992 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.591199 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.591200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.591200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.591261 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.591549 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.591550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.591602 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.591687 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.591805 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:27:05.591959 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.584711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.24.186.253:34059)
0928 20:27:05.584763 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.584766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.584782 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:27:05.584916 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.584916 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.584927 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.584995 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.585331 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.585915 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.586606 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.586823 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.586920 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.589098 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.589100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.589100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.589114 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.590680 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.591068 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.591069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.591070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.591131 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.591346 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.591346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.591347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.591479 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.591556 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.591726 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.591846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:27:05.592782 6239 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:27:05.593253 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.593134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41502)
0928 20:27:05.593188 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.593222 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 20:27:05.593771 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.593517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41516)
0928 20:27:05.593658 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.593721 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 20:27:05.593992 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
I20260928 20:27:05.609241 6314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 1
I20260928 20:27:05.609836 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.609690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41520)
0928 20:27:05.609763 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.609797 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 20:27:05.610319 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.610219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41522)
0928 20:27:05.610270 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.610295 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 20:27:05.610430 6314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
I20260928 20:27:05.625172 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 1
I20260928 20:27:05.625697 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.625553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41526)
0928 20:27:05.625623 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.625666 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 20:27:05.626061 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.625960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41542)
0928 20:27:05.626009 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.626035 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 20:27:05.626154 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
W20260928 20:27:05.663671 6321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:05.664013 6321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:05.664057 6321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:05.665930 6321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:05.666054 6321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:05.666083 6321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:05.666119 6321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:05.666164 6321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:05.668866 6321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:45063,127.24.186.253:34059,127.24.186.252:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:05.669219 6321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:05.669480 6321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:05.675709 6326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:05.675792 6327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:05.676069 6321 server_base.cc:1034] running on GCE node
W20260928 20:27:05.675792 6329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:05.676378 6321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:05.676709 6321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:05.677908 6321 hybrid_clock.cc:648] HybridClock initialized: now 1790627225677894 us; error 46 us; skew 500 ppm
I20260928 20:27:05.678581 6314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 2
I20260928 20:27:05.679030 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.678883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41550)
0928 20:27:05.678951 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.678993 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 20:27:05.679409 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.679307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41556)
0928 20:27:05.679356 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.679382 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:27:05.679497 6314 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:05.681272 6321 init.cc:377] Logged in from keytab as kudu/127.24.186.252@KRBTEST.COM (short username kudu)
I20260928 20:27:05.681916 6321 webserver.cc:460] Webserver started at http://127.24.186.252:40819/ using document root <none> and password file <none>
I20260928 20:27:05.682184 6321 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:05.682265 6321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:05.682416 6321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:05.683610 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data/instance:
uuid: "3e5646217bbf4bbcb85e4765af99e4fe"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "da026db99b97c3ff9aa32b9d853669e6"
server_key_iv: "15faf35eb5c911b71646976289d74df0"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.684120 6321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal/instance:
uuid: "3e5646217bbf4bbcb85e4765af99e4fe"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "da026db99b97c3ff9aa32b9d853669e6"
server_key_iv: "15faf35eb5c911b71646976289d74df0"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.685801 6321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:05.687036 6336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:05.687245 6321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:05.687371 6321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "3e5646217bbf4bbcb85e4765af99e4fe"
format_stamp: "Formatted at 2026-09-28 20:27:05 on dist-test-slave-k477"
server_key: "da026db99b97c3ff9aa32b9d853669e6"
server_key_iv: "15faf35eb5c911b71646976289d74df0"
server_key_version: "encryptionkey@0"
I20260928 20:27:05.687475 6321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:05.700019 6321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:05.701390 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 2
I20260928 20:27:05.702155 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.701874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41564)
0928 20:27:05.701969 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.702029 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 20:27:05.703006 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.702836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41570)
0928 20:27:05.702927 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.702969 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 20:27:05.703168 6239 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:38999: Network error: Client connection negotiation failed: client connection to 127.24.186.252:38999: connect: Connection refused (error 111)
I20260928 20:27:05.703459 6321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:05.703629 6321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:05.708173 6321 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.252:38999
I20260928 20:27:05.708592 6321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
I20260928 20:27:05.708784 6388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.252:38999 every 8 connection(s)
I20260928 20:27:05.709309 6389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:05.711004 6389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:05.718142 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6321
I20260928 20:27:05.718294 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790627060930553-25323-0/minicluster-data/master-2/wal/instance
I20260928 20:27:05.718538 25323 external_mini_cluster.cc:1428] Setting key f0284793b1bde9d5b08901b7af1c43cc
I20260928 20:27:05.721444 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.711774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.24.186.254:45063)
0928 20:27:05.711924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.711927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.712155 (+ 228us) server_negotiation.cc:407] Connection header received
0928 20:27:05.712708 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.712709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.712722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.712780 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.713476 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.714094 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.715168 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.715366 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.715456 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.717907 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.717908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.717908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.717916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.719926 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.720412 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.720413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.720414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.720480 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.720747 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.720748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.720748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.720900 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.720983 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.721264 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.721347 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 20:27:05.721673 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.711699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:45063 (local address 127.0.0.1:60178)
0928 20:27:05.712027 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.712066 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.712603 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.712857 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.712861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.713158 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.713370 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.713374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.714265 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.714266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.715071 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.715074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.715466 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.715467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.715601 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.716193 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.716219 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.717791 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.720026 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.720033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.720035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.720272 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.720559 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.720559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.720560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.720673 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.720986 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.720992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.721129 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.721243 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.721417 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":171,"threads_started":1}
I20260928 20:27:05.722242 6389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:27:05.728813 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.718976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.24.186.252:38999)
0928 20:27:05.719161 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.719165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.719210 (+ 45us) server_negotiation.cc:407] Connection header received
0928 20:27:05.719284 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.719286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.719298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.719361 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.720143 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.720856 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.721666 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.722096 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.722181 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.725037 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.725047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.725047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.725055 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.727033 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.727552 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.727556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.727557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.727621 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.727913 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.727914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.727915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.728128 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.728289 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.728567 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.728649 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260928 20:27:05.730281 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.722625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.24.186.253:34059)
0928 20:27:05.722673 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.722679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.722722 (+ 43us) server_negotiation.cc:407] Connection header received
0928 20:27:05.722787 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.722788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.722799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.722874 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.723111 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.723868 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.724790 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.725038 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.725101 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.727322 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.727323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.727324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.727332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.729112 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.729496 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.729497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.729497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.729553 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.729770 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.729771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.729771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.729917 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.729979 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.730149 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.730220 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:27:05.730346 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.722549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:34059 (local address 127.0.0.1:35634)
0928 20:27:05.722675 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.722687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.722734 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.722876 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.722877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.722993 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.723037 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.723038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.723933 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.723935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.724697 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.724699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.725156 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.725157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.725228 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.725760 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.725766 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.727235 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.729213 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.729214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.729215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.729435 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.729626 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.729627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.729627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.729685 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.730006 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.730007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.730051 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.730124 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.730222 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 20:27:05.730862 6389 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:27:05.737653 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.729848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.24.186.254:45063)
0928 20:27:05.729888 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.729890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.729899 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:27:05.729968 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.729969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.729977 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.730017 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.730616 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.731313 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.732140 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.732431 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.732497 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.734989 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.734990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.734990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.734997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.736486 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.736897 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.736898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.736899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.736964 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.737171 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.737171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.737172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.737279 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.737403 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.737519 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.737585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:27:05.738986 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.731104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41590)
0928 20:27:05.731150 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.731160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.731194 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.731318 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.731319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.731474 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.731507 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.731508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.732523 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.732524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.733241 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.733244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.733650 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.733652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.733721 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.734597 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.734603 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.735794 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.737858 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.737859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.737859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.738079 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.738292 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.738292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.738293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.738342 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.738584 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.738586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.738634 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.738708 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.738838 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 20:27:05.739084 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.731164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.24.186.252:38999)
0928 20:27:05.731195 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.731197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.731205 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:05.731244 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.731245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.731254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.731315 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.731608 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.732399 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.733356 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.733572 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.733626 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.735937 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.735939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.735939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.735947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.737763 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.738151 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.738152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.738152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.738217 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.738380 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.738380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.738381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.738514 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.738581 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.738746 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.738913 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":16}
I20260928 20:27:05.740736 6389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: Bootstrap starting.
I20260928 20:27:05.741844 6389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:05.742175 6389 log.cc:824] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:05.743551 6389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: No bootstrap required, opened a new log
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:27:05.745463 6389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.745620 6389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:05.745668 6389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e5646217bbf4bbcb85e4765af99e4fe, State: Initialized, Role: FOLLOWER
I20260928 20:27:05.745839 6389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.746228 6389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:05.746187 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.746378 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:27:05.748201 6406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:27:05.748260 6406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:27:05.748613 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.738853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.24.186.253:34059)
0928 20:27:05.738895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.738897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.738905 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:05.738930 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.738931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.738942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.739021 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.740082 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.740548 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.741598 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.741853 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.742024 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.745352 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.745354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.745354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.745371 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.747238 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.747646 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.747648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.747648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.747739 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.748028 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.748029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.748029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.748183 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.748367 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.748426 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.748511 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:27:05.750507 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:27:05.748675 6321 master_runner.cc:186] Error getting master registration for 127.24.186.254:45063: OK, instance_id { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" instance_seqno: 1790627225352169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:27:05.751142 6321 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:27:05.759080 6318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.752063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.24.186.253:34059)
0928 20:27:05.752104 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.752106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.752403 (+ 297us) server_negotiation.cc:407] Connection header received
0928 20:27:05.752482 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.752483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.752493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.752537 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.753072 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.753541 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.754248 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.754455 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.754541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.755836 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.755837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.755838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.755844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.757752 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.758198 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.758200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.758200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.758261 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.758475 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.758476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.758476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.758609 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.758678 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.758883 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.758978 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:27:05.764245 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.752351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.24.186.254:45063)
0928 20:27:05.752402 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.752404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.756031 (+ 3627us) server_negotiation.cc:407] Connection header received
0928 20:27:05.756086 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.756087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.756101 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.756155 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.756870 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.757789 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.758929 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.759355 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.759424 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.761513 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.761514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.761515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.761523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.762921 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.763286 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.763288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.763288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.763350 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.763571 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.763572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.763572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.763725 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.763839 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.764053 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.764132 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:27:05.764648 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.752300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.24.186.252:38999)
0928 20:27:05.752339 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.752341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.756164 (+ 3823us) server_negotiation.cc:407] Connection header received
0928 20:27:05.756265 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.756266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.756276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.756305 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.757024 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.757754 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.758937 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.759416 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.759461 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.761218 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.761220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.761220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.761229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.762921 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.763581 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.763582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.763583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.763647 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.763882 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.763883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.763884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.764083 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.764280 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.764468 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.764555 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:27:05.770649 6314 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 3
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:27:05.779030 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.771043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41618)
0928 20:27:05.771112 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.771124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.771184 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.771441 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.771442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.771590 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.771636 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.771638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.772623 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.772624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.773130 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.773132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.773637 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.773639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.773685 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.774173 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.774179 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.775388 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.778037 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.778038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.778039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.778219 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.778432 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.778433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.778433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.778485 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.778763 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.778764 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.778807 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.778867 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.778958 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 20:27:05.779098 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.771252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.24.186.252:38999)
0928 20:27:05.771318 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.771320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.771330 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:05.771355 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.771356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.771367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.771429 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.771709 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.772524 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.773212 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.773513 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.773585 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.775469 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.775471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.775471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.775478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.777931 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.778294 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.778296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.778296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.778361 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.778544 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.778545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.778546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.778695 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.778763 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.778913 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.779002 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:27:05.780829 6314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: Bootstrap starting.
I20260928 20:27:05.781527 6314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:05.781841 6314 log.cc:824] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:05.782738 6314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: No bootstrap required, opened a new log
I20260928 20:27:05.784729 6314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.784873 6314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:05.784922 6314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81dd0f4ec2b4d548b8aed49d251b456, State: Initialized, Role: FOLLOWER
I20260928 20:27:05.785105 6314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.785395 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.785470 6314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:05.785517 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:27:05.787735 6427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:27:05.787864 6427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:27:05.788199 6246 master_runner.cc:186] Error getting master registration for 127.24.186.254:45063: OK, instance_id { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" instance_seqno: 1790627225352169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:27:05.791922 6246 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:27:05.802359 6239 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } attempt: 3
Sep 28 20:27:05 dist-test-slave-k477 krb5kdc[6158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627225, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:27:05.812184 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.802790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.24.186.252:38999)
0928 20:27:05.802850 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:05.802852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:05.802862 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:05.802892 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:05.802893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:05.802904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:05.802974 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:05.803315 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.804097 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.805237 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.805619 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.805765 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.808286 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:05.808288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:05.808288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:05.808298 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:05.810329 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.811131 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.811132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.811132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.811207 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:05.811454 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:05.811455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:05.811455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:05.811611 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:05.811682 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:05.811951 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 20:27:05.812040 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:27:05.812176 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:05.802728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:38999 (local address 127.0.0.1:41628)
0928 20:27:05.802782 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:05.802794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:05.802833 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:05.802992 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:05.802993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:05.803184 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:05.803239 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.803241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.804194 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.804195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.805067 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:05.805071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:05.805770 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:05.805771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:05.805837 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:05.806486 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:05.806492 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:05.808167 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:05.810461 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.810463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.810464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.811020 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.811290 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:05.811291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:05.811291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:05.811380 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:05.811688 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:05.811690 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:05.811750 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 20:27:05.811886 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 20:27:05.812004 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:27:05.815455 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: Bootstrap starting.
I20260928 20:27:05.816555 6239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:05.816987 6239 log.cc:824] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:05.818085 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: No bootstrap required, opened a new log
I20260928 20:27:05.821341 6239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.821544 6239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:05.821583 6239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383277af132c4187aba71dc3f8a25c44, State: Initialized, Role: FOLLOWER
I20260928 20:27:05.821745 6239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.822139 6239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:05.822090 6430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.823570 6430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:27:05.824195 6441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:27:05.824306 6441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:27:05.826638 6171 master_runner.cc:263] No leader master found from master 383277af132c4187aba71dc3f8a25c44
I20260928 20:27:05.826689 6171 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 20:27:05.850354 6192 tablet.cc:2369] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:27:05.930459 6395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:05.930681 6395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.931301 6395 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 383277af132c4187aba71dc3f8a25c44 (127.24.186.254:45063), d81dd0f4ec2b4d548b8aed49d251b456 (127.24.186.253:34059)
I20260928 20:27:05.931614 6213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383277af132c4187aba71dc3f8a25c44" is_pre_election: true
I20260928 20:27:05.931826 6213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5646217bbf4bbcb85e4765af99e4fe in term 0.
I20260928 20:27:05.931789 6288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" is_pre_election: true
I20260928 20:27:05.931959 6288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e5646217bbf4bbcb85e4765af99e4fe in term 0.
I20260928 20:27:05.932020 6339 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383277af132c4187aba71dc3f8a25c44, 3e5646217bbf4bbcb85e4765af99e4fe; no voters:
I20260928 20:27:05.932207 6395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:05.932253 6395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:05.932273 6395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:05.933039 6395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.933204 6395 leader_election.cc:288] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [CANDIDATE]: Term 1 election: Requested vote from peers 383277af132c4187aba71dc3f8a25c44 (127.24.186.254:45063), d81dd0f4ec2b4d548b8aed49d251b456 (127.24.186.253:34059)
I20260928 20:27:05.933394 6213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "383277af132c4187aba71dc3f8a25c44"
I20260928 20:27:05.933388 6288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81dd0f4ec2b4d548b8aed49d251b456"
I20260928 20:27:05.933458 6288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:05.933458 6213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:05.934312 6288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5646217bbf4bbcb85e4765af99e4fe in term 1.
I20260928 20:27:05.934312 6213 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e5646217bbf4bbcb85e4765af99e4fe in term 1.
I20260928 20:27:05.934468 6339 leader_election.cc:302] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 383277af132c4187aba71dc3f8a25c44, 3e5646217bbf4bbcb85e4765af99e4fe; no voters:
I20260928 20:27:05.934552 6395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:05.934741 6395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [term 1 LEADER]: Becoming Leader. State: Replica: 3e5646217bbf4bbcb85e4765af99e4fe, State: Running, Role: LEADER
I20260928 20:27:05.934840 6395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } }
I20260928 20:27:05.935238 6443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5646217bbf4bbcb85e4765af99e4fe. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.935316 6443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:05.935494 6445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:05.935686 6445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:05.936969 6213 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [term 1 FOLLOWER]: Refusing update from remote peer 3e5646217bbf4bbcb85e4765af99e4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:05.936990 6288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [term 1 FOLLOWER]: Refusing update from remote peer 3e5646217bbf4bbcb85e4765af99e4fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:05.937252 6443 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:05.937337 6395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:05.938545 6430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5646217bbf4bbcb85e4765af99e4fe. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.938551 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e5646217bbf4bbcb85e4765af99e4fe. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.938633 6430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:27:05.938633 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:27:05.939590 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.939672 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:05.939709 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.939751 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:05.939778 6430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.939873 6430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:27:05.940488 6416 sys_catalog.cc:455] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "383277af132c4187aba71dc3f8a25c44" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 45063 } } peers { permanent_uuid: "d81dd0f4ec2b4d548b8aed49d251b456" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 34059 } } peers { permanent_uuid: "3e5646217bbf4bbcb85e4765af99e4fe" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 38999 } } }
I20260928 20:27:05.940552 6416 sys_catalog.cc:458] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:27:05.940939 6446 mvcc.cc:205] Tried to move back new op lower bound from 7334409117435834368 to 7334409117429723136. Current Snapshot: MvccSnapshot[applied={T|T < 7334409117435834368}]
I20260928 20:27:05.941452 6445 catalog_manager.cc:1347] Generated new cluster ID: c0a1febbe3c54522af33348536235046
I20260928 20:27:05.941505 6445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:27:05.941906 6448 mvcc.cc:205] Tried to move back new op lower bound from 7334409117435834368 to 7334409117429723136. Current Snapshot: MvccSnapshot[applied={T|T < 7334409117435834368}]
I20260928 20:27:05.942109 6447 mvcc.cc:205] Tried to move back new op lower bound from 7334409117435834368 to 7334409117429723136. Current Snapshot: MvccSnapshot[applied={T|T < 7334409117435834368}]
I20260928 20:27:05.958611 6445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:27:05.959256 6445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:05.962471 6445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e5646217bbf4bbcb85e4765af99e4fe: Generated new TSK 0
I20260928 20:27:05.962683 6445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:27:06.788321 6427 catalog_manager.cc:1259] Loaded cluster ID: c0a1febbe3c54522af33348536235046
I20260928 20:27:06.788401 6427 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: loading cluster ID for follower catalog manager: success
I20260928 20:27:06.789275 6427 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: acquiring CA information for follower catalog manager: success
I20260928 20:27:06.789587 6427 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d81dd0f4ec2b4d548b8aed49d251b456: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:27:06.824779 6441 catalog_manager.cc:1259] Loaded cluster ID: c0a1febbe3c54522af33348536235046
I20260928 20:27:06.824858 6441 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: loading cluster ID for follower catalog manager: success
I20260928 20:27:06.825709 6441 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: acquiring CA information for follower catalog manager: success
I20260928 20:27:06.826031 6441 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 383277af132c4187aba71dc3f8a25c44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:27:07.028663 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6171
I20260928 20:27:07.034102 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6246
I20260928 20:27:07.039773 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6321
2026-09-28T20:27:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3055 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:27:07 dist-test-slave-k477 krb5kdc[6460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:27:07 dist-test-slave-k477 krb5kdc[6460](info): set up 2 sockets
Sep 28 20:27:07 dist-test-slave-k477 krb5kdc[6460](info): commencing operation
krb5kdc: starting...
W20260928 20:27:08.113133 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:27:08.234586 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40973
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:34077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:08.315691 6473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:08.315985 6473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:08.316030 6473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:08.317189 6473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:08.317263 6473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:08.317287 6473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:08.317314 6473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:08.317345 6473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:08.318820 6473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:34077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:08.319110 6473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:08.319342 6473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:08.324165 6479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:08.324185 6481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:08.324165 6478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:08.324448 6473 server_base.cc:1034] running on GCE node
I20260928 20:27:08.324720 6473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:08.325030 6473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:08.326223 6473 hybrid_clock.cc:648] HybridClock initialized: now 1790627228326207 us; error 45 us; skew 500 ppm
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627228, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:08.337500 6473 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:08.338014 6473 webserver.cc:460] Webserver started at http://127.24.186.254:35133/ using document root <none> and password file <none>
I20260928 20:27:08.338204 6473 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:08.338245 6473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:08.338336 6473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:08.339368 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "b35654b0ee78436b8ffbf7c762f19c92"
format_stamp: "Formatted at 2026-09-28 20:27:08 on dist-test-slave-k477"
server_key: "4a5c6b7bffe4ef8e9310ff6d0d89960d"
server_key_iv: "9b8c44b9e90e591e0170bf4408707d12"
server_key_version: "encryptionkey@0"
I20260928 20:27:08.339674 6473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "b35654b0ee78436b8ffbf7c762f19c92"
format_stamp: "Formatted at 2026-09-28 20:27:08 on dist-test-slave-k477"
server_key: "4a5c6b7bffe4ef8e9310ff6d0d89960d"
server_key_iv: "9b8c44b9e90e591e0170bf4408707d12"
server_key_version: "encryptionkey@0"
I20260928 20:27:08.341150 6473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:08.341943 6488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:08.342146 6473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:08.342228 6473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "b35654b0ee78436b8ffbf7c762f19c92"
format_stamp: "Formatted at 2026-09-28 20:27:08 on dist-test-slave-k477"
server_key: "4a5c6b7bffe4ef8e9310ff6d0d89960d"
server_key_iv: "9b8c44b9e90e591e0170bf4408707d12"
server_key_version: "encryptionkey@0"
I20260928 20:27:08.342288 6473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:08.354665 6473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:08.357049 6473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:08.357167 6473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:08.361277 6473 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:40973
I20260928 20:27:08.361339 6540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:40973 every 8 connection(s)
I20260928 20:27:08.361687 6473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:08.362565 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:08.364823 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92: Bootstrap starting.
I20260928 20:27:08.365383 6541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:08.365617 6541 log.cc:824] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:08.366261 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92: No bootstrap required, opened a new log
I20260928 20:27:08.367779 6541 raft_consensus.cc:348] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } }
I20260928 20:27:08.368052 6541 raft_consensus.cc:374] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:08.368081 6541 raft_consensus.cc:729] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35654b0ee78436b8ffbf7c762f19c92, State: Initialized, Role: FOLLOWER
I20260928 20:27:08.368193 6541 consensus_queue.cc:260] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } }
I20260928 20:27:08.368276 6541 raft_consensus.cc:388] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:08.368315 6541 raft_consensus.cc:482] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:08.368367 6541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:08.368909 6541 raft_consensus.cc:504] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } }
I20260928 20:27:08.369032 6541 leader_election.cc:302] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b35654b0ee78436b8ffbf7c762f19c92; no voters:
I20260928 20:27:08.369217 6541 leader_election.cc:288] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:27:08.369259 6544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:08.369422 6544 raft_consensus.cc:686] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [term 1 LEADER]: Becoming Leader. State: Replica: b35654b0ee78436b8ffbf7c762f19c92, State: Running, Role: LEADER
I20260928 20:27:08.369489 6541 sys_catalog.cc:564] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:08.369607 6544 consensus_queue.cc:237] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } }
I20260928 20:27:08.369899 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6473
I20260928 20:27:08.370009 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:27:08.370028 6546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35654b0ee78436b8ffbf7c762f19c92. Latest consensus state: current_term: 1 leader_uuid: "b35654b0ee78436b8ffbf7c762f19c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } } }
I20260928 20:27:08.370102 6546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:08.370131 6545 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b35654b0ee78436b8ffbf7c762f19c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b35654b0ee78436b8ffbf7c762f19c92" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40973 } } }
I20260928 20:27:08.370184 6545 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92 [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:08.370241 25323 external_mini_cluster.cc:1428] Setting key 60764151d5cec5a4b93ad54727a3bc27
I20260928 20:27:08.370314 6551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:08.370831 6551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:08.372604 6551 catalog_manager.cc:1347] Generated new cluster ID: c4a095836cfe476dbb4614c018c372e1
I20260928 20:27:08.372681 6551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:08.382483 6561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:08.370855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.24.186.254:40973)
0928 20:27:08.371080 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:08.371089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:08.371380 (+ 291us) server_negotiation.cc:407] Connection header received
0928 20:27:08.371502 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:08.371509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:08.371567 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:08.371712 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:08.372624 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:08.373982 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:08.375037 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:08.375736 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:08.375877 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:08.378756 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:08.378760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:08.378760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:08.378772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:08.380764 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:08.381186 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:08.381189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:08.381189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:08.381303 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:08.381519 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:08.381519 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:08.381520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:08.381828 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:08.381966 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:08.382111 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 20:27:08.382244 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":61,"threads_started":1}
I20260928 20:27:08.384196 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:27:08.408263 6551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:27:08.408839 6551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:08.411374 6551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b35654b0ee78436b8ffbf7c762f19c92: Generated new TSK 0
I20260928 20:27:08.411597 6551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:27:08.438688 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6473
2026-09-28T20:27:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1394 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6568](info): set up 2 sockets
Sep 28 20:27:08 dist-test-slave-k477 krb5kdc[6568](info): commencing operation
krb5kdc: starting...
W20260928 20:27:09.504731 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:27:09 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:27:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:27:09.630432 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:09.714241 6581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:09.714473 6581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:09.714496 6581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:09.715601 6581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:09.715638 6581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:09.715653 6581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:09.715665 6581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:09.715677 6581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:09.717130 6581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:09.717337 6581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:09.717525 6581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:09.722465 6586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:09.722481 6587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:09.722625 6581 server_base.cc:1034] running on GCE node
W20260928 20:27:09.722476 6589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:09.723201 6581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:09.723531 6581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:09.724732 6581 hybrid_clock.cc:648] HybridClock initialized: now 1790627229724725 us; error 36 us; skew 500 ppm
Sep 28 20:27:09 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:09.728346 6581 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:09.728808 6581 webserver.cc:460] Webserver started at http://127.24.186.254:44097/ using document root <none> and password file <none>
I20260928 20:27:09.729055 6581 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:09.729126 6581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:09.729274 6581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:09.730360 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "a30f4b9f34fdb21fa3fc78225d1df81e"
server_key_iv: "95b3062bf7a6c11b2b9e1772922de9ee"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.730727 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "a30f4b9f34fdb21fa3fc78225d1df81e"
server_key_iv: "95b3062bf7a6c11b2b9e1772922de9ee"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.732174 6581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:27:09.732998 6596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:09.733268 6581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:09.733381 6581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "a30f4b9f34fdb21fa3fc78225d1df81e"
server_key_iv: "95b3062bf7a6c11b2b9e1772922de9ee"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.733476 6581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:09.751708 6581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:09.753612 6581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:09.753777 6581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:09.758685 6581 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38383
I20260928 20:27:09.758709 6648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38383 every 8 connection(s)
I20260928 20:27:09.759194 6581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:09.760247 6649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:09.762557 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap starting.
I20260928 20:27:09.763159 6649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:09.763448 6649 log.cc:824] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:09.764243 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: No bootstrap required, opened a new log
I20260928 20:27:09.765898 6649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:09.766043 6649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:09.766067 6649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Initialized, Role: FOLLOWER
I20260928 20:27:09.766234 6649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:09.766302 6649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:09.766323 6649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:09.766359 6649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:09.766480 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6581
I20260928 20:27:09.766569 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:27:09.766708 25323 external_mini_cluster.cc:1428] Setting key 892561b51ed7983589d652087737d234
I20260928 20:27:09.767341 6649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:09.767478 6649 leader_election.cc:302] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b5fa76e5f0e4cfe8d8ab653b6de079e; no voters:
I20260928 20:27:09.767670 6649 leader_election.cc:288] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:27:09.767706 6654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:09.767854 6654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 LEADER]: Becoming Leader. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Running, Role: LEADER
I20260928 20:27:09.767963 6649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:09.768009 6654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:09.768371 6654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b5fa76e5f0e4cfe8d8ab653b6de079e. Latest consensus state: current_term: 1 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:09.768465 6655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:09.768518 6654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:09.768559 6655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:09.768743 6661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:09.769302 6661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:09.770952 6661 catalog_manager.cc:1347] Generated new cluster ID: 37df689d7c174bc4b8250b1db01acba6
I20260928 20:27:09.771003 6661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:27:09 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:09.777583 6661 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:27:09.778288 6661 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:09.779239 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:09.767229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.24.186.254:38383)
0928 20:27:09.767410 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:09.767449 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:09.767485 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:27:09.767614 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:09.767621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:09.767700 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:09.767924 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:09.768847 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.770294 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.771361 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.771990 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.772094 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:09.774950 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:09.774954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:09.774954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:09.774966 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:09.777456 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:09.777890 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:09.777894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:09.777894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:09.778011 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:09.778252 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:09.778253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:09.778254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:09.778552 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:09.778709 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:09.778827 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 20:27:09.779008 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260928 20:27:09.786922 6661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Generated new TSK 0
I20260928 20:27:09.787141 6661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:27:09.816229 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:09.903192 6675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:09.903416 6675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:09.903445 6675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:09.904644 6675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:09.904685 6675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:09.904711 6675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:09.906672 6675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:09.906914 6675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:09.907115 6675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:09.912171 6680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:09.912178 6681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:09.912287 6675 server_base.cc:1034] running on GCE node
W20260928 20:27:09.912195 6683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:09.912647 6675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:09.912973 6675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:09.914175 6675 hybrid_clock.cc:648] HybridClock initialized: now 1790627229914164 us; error 29 us; skew 500 ppm
Sep 28 20:27:09 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:09.916954 6675 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:09.917412 6675 webserver.cc:460] Webserver started at http://127.24.186.193:36249/ using document root <none> and password file <none>
I20260928 20:27:09.917663 6675 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:09.917763 6675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:09.917924 6675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:09.919016 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "b8f5239fd0144285b5eb27c32860b09d"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "6c8366f3bd799de0029afd925c0e1655"
server_key_iv: "be2dfe51a76d877ab50970fc611a97d7"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.919389 6675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "b8f5239fd0144285b5eb27c32860b09d"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "6c8366f3bd799de0029afd925c0e1655"
server_key_iv: "be2dfe51a76d877ab50970fc611a97d7"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.921084 6675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:27:09.921938 6690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:09.922199 6675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:09.922310 6675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b8f5239fd0144285b5eb27c32860b09d"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "6c8366f3bd799de0029afd925c0e1655"
server_key_iv: "be2dfe51a76d877ab50970fc611a97d7"
server_key_version: "encryptionkey@0"
I20260928 20:27:09.922413 6675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:09.942629 6675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:09.944778 6675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:09.944953 6675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:09.945195 6675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:09.945511 6675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:09.945564 6675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:09.945606 6675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:09.945634 6675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:09.951788 6675 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:39215
I20260928 20:27:09.951861 6803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:39215 every 8 connection(s)
I20260928 20:27:09.952194 6675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:09.952616 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6675
I20260928 20:27:09.952745 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:27:09.952951 25323 external_mini_cluster.cc:1428] Setting key 46a94cd99753b7ca28b0d7b876243c7f
Sep 28 20:27:09 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627229, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:27:09.963644 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:09.952973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:50887 (local address 127.24.186.254:38383)
0928 20:27:09.953051 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:09.953053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:09.953316 (+ 263us) server_negotiation.cc:407] Connection header received
0928 20:27:09.953907 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:09.953908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:09.953921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:09.953999 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:09.954962 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.955584 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.956834 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.957195 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.957271 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:09.960097 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:09.960100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:09.960101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:09.960116 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:09.962011 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:09.962447 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:09.962449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:09.962449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:09.962521 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:09.962886 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:09.962887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:09.962887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:09.963035 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:09.963097 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:09.963457 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 20:27:09.963574 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 20:27:09.963891 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:09.952976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:50887)
0928 20:27:09.953170 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:09.953224 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:09.953787 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:09.954060 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:09.954064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:09.954649 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:09.954863 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:09.954868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.955689 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.955690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:09.956742 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:09.956744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:09.957289 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:09.957290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:09.957400 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:09.957995 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:09.958006 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:09.959953 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:09.962113 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:09.962119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:09.962121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:09.962352 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:09.962653 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:09.962655 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:09.962655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:09.962803 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:09.963102 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:09.963107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:09.963234 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 20:27:09.963323 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:27:09.963540 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260928 20:27:09.964439 6804 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:09.964560 6804 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:09.964752 6804 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:09.965337 6613 ts_manager.cc:194] Registered new tserver with Master: b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
I20260928 20:27:09.966065 6613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:50887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:27:09.984129 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:10.073100 6809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.073333 6809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.073359 6809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.074656 6809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.074714 6809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.074743 6809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:10.076438 6809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.076997 6809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.077276 6809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:27:10.082810 6809 server_base.cc:1034] running on GCE node
W20260928 20:27:10.082867 6814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.082751 6815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.082760 6817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.083274 6809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.083961 6809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.085661 6809 hybrid_clock.cc:648] HybridClock initialized: now 1790627230085638 us; error 40 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.089411 6809 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:10.090008 6809 webserver.cc:460] Webserver started at http://127.24.186.194:37137/ using document root <none> and password file <none>
I20260928 20:27:10.090255 6809 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.090325 6809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.090488 6809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:10.091599 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "caf84d20b28b46ed9d90060bbc716535"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "93963f72723539e213cf18f608a1619a"
server_key_iv: "0b6fcdbdf178f9b718c67c2d160bbd71"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.091992 6809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "caf84d20b28b46ed9d90060bbc716535"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "93963f72723539e213cf18f608a1619a"
server_key_iv: "0b6fcdbdf178f9b718c67c2d160bbd71"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.093354 6809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:10.094143 6824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.094342 6809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:27:10.094440 6809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "caf84d20b28b46ed9d90060bbc716535"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "93963f72723539e213cf18f608a1619a"
server_key_iv: "0b6fcdbdf178f9b718c67c2d160bbd71"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.094527 6809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.111239 6809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.113327 6809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.113502 6809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.113745 6809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:10.114073 6809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:10.114127 6809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.114161 6809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:10.114189 6809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.121249 6809 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:39131
I20260928 20:27:10.121289 6937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:39131 every 8 connection(s)
I20260928 20:27:10.121663 6809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:10.130594 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6809
I20260928 20:27:10.130721 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:27:10.130932 25323 external_mini_cluster.cc:1428] Setting key b9bc1558581f13c839e532dc228b4bb0
I20260928 20:27:10.133093 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.122472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:42797)
0928 20:27:10.122645 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.122689 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.123181 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.123387 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.123390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.124055 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.124249 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.124254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.124967 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.124967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.125880 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.125882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.126706 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.126706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.126835 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.127447 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.127456 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.129445 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.131481 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.131490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.131493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.131903 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.132145 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.132145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.132146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.132256 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.132568 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.132573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.132702 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.132782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.132912 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260928 20:27:10.133095 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.122520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:42797 (local address 127.24.186.254:38383)
0928 20:27:10.122606 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.122609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.122774 (+ 165us) server_negotiation.cc:407] Connection header received
0928 20:27:10.123260 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.123261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.123271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.123334 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.124341 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.124845 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.126076 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.126481 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.126586 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.129562 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.129564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.129564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.129574 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.131386 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.131995 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.131996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.131997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.132070 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.132332 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.132333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.132333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.132494 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.132576 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.132840 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.132953 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 20:27:10.133893 6938 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:10.134022 6938 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:10.134212 6938 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:10.134639 6613 ts_manager.cc:194] Registered new tserver with Master: caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:10.135164 6613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:42797
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:27:10.159564 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:10.247615 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.247923 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.247956 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.249182 6943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.249230 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.249258 6943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:10.250748 6943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.251006 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.251248 6943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:10.256672 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.256742 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.256764 6943 server_base.cc:1034] running on GCE node
W20260928 20:27:10.256673 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.257122 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.257429 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.258602 6943 hybrid_clock.cc:648] HybridClock initialized: now 1790627230258637 us; error 82 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.261181 6943 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:10.261672 6943 webserver.cc:460] Webserver started at http://127.24.186.195:46047/ using document root <none> and password file <none>
I20260928 20:27:10.261914 6943 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.261992 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.262141 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:10.263216 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "f83e217fe36241e5a4df6ef68bca546c"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "9afa77d052df54e1d29a27a9380f19af"
server_key_iv: "11feb1702fa220976b32e06c7e57f5e6"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.263593 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "f83e217fe36241e5a4df6ef68bca546c"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "9afa77d052df54e1d29a27a9380f19af"
server_key_iv: "11feb1702fa220976b32e06c7e57f5e6"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.265448 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 20:27:10.266707 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.266980 6943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:27:10.267119 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "f83e217fe36241e5a4df6ef68bca546c"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "9afa77d052df54e1d29a27a9380f19af"
server_key_iv: "11feb1702fa220976b32e06c7e57f5e6"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.267212 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.279295 6943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.281379 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.281559 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.281801 6943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:10.282124 6943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:10.282178 6943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.282222 6943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:10.282248 6943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.288820 6943 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:36547
I20260928 20:27:10.288856 7071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:36547 every 8 connection(s)
I20260928 20:27:10.289206 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:10.295948 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 6943
I20260928 20:27:10.296052 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:27:10.296244 25323 external_mini_cluster.cc:1428] Setting key b0d05dfa78f57ecbf8b00d8312253385
I20260928 20:27:10.299180 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.289960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:58229 (local address 127.24.186.254:38383)
0928 20:27:10.290026 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.290028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.290257 (+ 229us) server_negotiation.cc:407] Connection header received
0928 20:27:10.290722 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.290724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.290734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.290782 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.291695 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.292252 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.293252 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.293447 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.293504 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.295976 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.295978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.295978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.295988 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.297915 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.298363 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.298363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.298364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.298415 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.298679 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.298679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.298680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.298792 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.298846 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.299052 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.299113 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:27:10.299444 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.289948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:58229)
0928 20:27:10.290118 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.290155 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.290636 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.290843 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.290846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.291416 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.291611 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.291616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.292364 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.292365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.293171 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.293173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.293557 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.293558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.293685 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.294218 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.294227 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.295858 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.298010 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.298020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.298023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.298290 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.298514 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.298515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.298515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.298619 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.298836 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.298839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.298961 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.299042 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.299190 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260928 20:27:10.299981 7072 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:10.300104 7072 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:10.300293 7072 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:10.300684 6613 ts_manager.cc:194] Registered new tserver with Master: f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547)
I20260928 20:27:10.301086 6613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:58229
I20260928 20:27:10.308624 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:27:10.316521 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.310059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.24.186.254:38383)
0928 20:27:10.310145 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.310148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.310264 (+ 116us) server_negotiation.cc:407] Connection header received
0928 20:27:10.310356 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.310357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.310368 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.310435 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.310983 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.311482 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.312260 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.312475 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.312562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.313724 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.313725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.313726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.313733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.315240 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.315612 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.315614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.315614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.315671 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.315943 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.315944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.315945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.316106 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.316219 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.316323 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.316414 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 20:27:10.318207 6613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:27:10.319795 6613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:27:10.326705 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.24.186.193:39215)
0928 20:27:10.322442 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.322453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.322465 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:27:10.322508 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.322511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.322527 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.322615 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:10.322868 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.324079 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.325703 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326243 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326331 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326388 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.326566 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.326605 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
I20260928 20:27:10.326735 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39215 (local address 127.0.0.1:38606)
0928 20:27:10.322218 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.322258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.322321 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.322707 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.322710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.322713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:10.322793 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.322795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.324167 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.324168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.325568 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.325571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326330 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.326430 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326444 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.326547 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.326606 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":112,"thread_start_us":69,"threads_started":1}
I20260928 20:27:10.326752 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36547 (local address 127.0.0.1:35702)
0928 20:27:10.322619 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.322630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.322692 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.323042 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.323043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.323044 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:10.323091 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.323093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.324106 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.324107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.325599 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.325600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326486 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.326555 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326561 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.326612 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.326655 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":93,"thread_start_us":83,"threads_started":1}
I20260928 20:27:10.326830 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.24.186.194:39131)
0928 20:27:10.322773 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.322777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.322941 (+ 164us) server_negotiation.cc:407] Connection header received
0928 20:27:10.322982 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.322985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.323000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.323146 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:10.323338 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.324366 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.325594 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326301 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326445 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.326700 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.326742 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260928 20:27:10.326980 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.24.186.195:36547)
0928 20:27:10.322637 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.322642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.322680 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:27:10.322762 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.322764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.322780 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.322970 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:10.323196 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.323998 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.325660 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326336 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326402 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326445 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.326850 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.326910 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260928 20:27:10.327064 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.322476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.0.0.1:56902)
0928 20:27:10.322825 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.322835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.322883 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.323216 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.323216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.323218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:10.323253 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.323254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.324510 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.324510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.325521 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.325524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.326484 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.326486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.326555 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.326562 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.326612 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.326655 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":6,"thread_start_us":69,"threads_started":1}
I20260928 20:27:10.327838 7006 tablet_service.cc:1467] Processing CreateTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd (DEFAULT_TABLE table=test-table [id=f3ba74060a574bcc98382130cee1a9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:10.327701 6872 tablet_service.cc:1467] Processing CreateTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd (DEFAULT_TABLE table=test-table [id=f3ba74060a574bcc98382130cee1a9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:10.328239 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68837d6fdcb4038ae61ce16fd0fb2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:10.328464 6872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68837d6fdcb4038ae61ce16fd0fb2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:10.328642 6738 tablet_service.cc:1467] Processing CreateTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd (DEFAULT_TABLE table=test-table [id=f3ba74060a574bcc98382130cee1a9aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:10.329144 6738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d68837d6fdcb4038ae61ce16fd0fb2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:10.330734 7089 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Bootstrap starting.
I20260928 20:27:10.331354 7089 tablet_bootstrap.cc:654] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:10.331777 7089 log.cc:824] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:10.332391 7091 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Bootstrap starting.
I20260928 20:27:10.332441 7092 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Bootstrap starting.
I20260928 20:27:10.333043 7089 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: No bootstrap required, opened a new log
I20260928 20:27:10.333081 7091 tablet_bootstrap.cc:654] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:10.333127 7092 tablet_bootstrap.cc:654] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:10.333140 7089 ts_tablet_manager.cc:1397] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:27:10.333737 7091 log.cc:824] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:10.333791 7092 log.cc:824] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:10.334510 7091 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: No bootstrap required, opened a new log
I20260928 20:27:10.334671 7091 ts_tablet_manager.cc:1397] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:10.334710 7092 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: No bootstrap required, opened a new log
I20260928 20:27:10.334785 7092 ts_tablet_manager.cc:1397] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:10.335122 7089 raft_consensus.cc:348] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.335279 7089 raft_consensus.cc:374] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:10.335336 7089 raft_consensus.cc:729] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83e217fe36241e5a4df6ef68bca546c, State: Initialized, Role: FOLLOWER
I20260928 20:27:10.335465 7089 consensus_queue.cc:260] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.335762 7089 ts_tablet_manager.cc:1428] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:27:10.335937 7072 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
I20260928 20:27:10.336799 7091 raft_consensus.cc:348] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.336807 7092 raft_consensus.cc:348] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.336957 7092 raft_consensus.cc:374] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:10.336957 7091 raft_consensus.cc:374] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:10.337001 7091 raft_consensus.cc:729] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf84d20b28b46ed9d90060bbc716535, State: Initialized, Role: FOLLOWER
I20260928 20:27:10.337028 7092 raft_consensus.cc:729] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5239fd0144285b5eb27c32860b09d, State: Initialized, Role: FOLLOWER
I20260928 20:27:10.337121 7091 consensus_queue.cc:260] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.337164 7092 consensus_queue.cc:260] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.337409 7091 ts_tablet_manager.cc:1428] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:27:10.337407 7092 ts_tablet_manager.cc:1428] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:27:10.337412 6804 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
I20260928 20:27:10.337457 6938 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
I20260928 20:27:10.337731 7095 raft_consensus.cc:482] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:10.337818 7095 raft_consensus.cc:504] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.338141 7095 leader_election.cc:288] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:10.342286 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.338614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:60175 (local address 127.24.186.193:39215)
0928 20:27:10.338652 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.338654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.338661 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:27:10.338680 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.338681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.338689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.338737 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:10.338893 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.339615 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341276 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.341873 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341957 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.341966 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.342182 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.342220 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:27:10.342286 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.338441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:43873 (local address 127.24.186.194:39131)
0928 20:27:10.338484 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.338486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.338602 (+ 116us) server_negotiation.cc:407] Connection header received
0928 20:27:10.338650 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.338651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.338661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.338716 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:10.338878 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.339615 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341276 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.341873 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341959 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.341966 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.342182 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.342220 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:27:10.342389 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.338274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39215 (local address 127.24.186.195:60175)
0928 20:27:10.338314 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.338327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.338368 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.338776 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.338777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.338778 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:10.338822 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.338823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.339778 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.339779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.341173 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.341176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.341991 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.342097 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.342103 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.342163 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.342218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":159}
I20260928 20:27:10.342700 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.338371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.24.186.195:43873)
0928 20:27:10.338525 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.338537 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.338594 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.338716 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.338717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.338719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:10.338794 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.338796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.339778 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.339779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.341174 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.341176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.341987 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.341988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.342097 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.342103 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.342163 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.342218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":67,"thread_start_us":107,"threads_started":1}
I20260928 20:27:10.342764 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68837d6fdcb4038ae61ce16fd0fb2bd" candidate_uuid: "f83e217fe36241e5a4df6ef68bca546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5239fd0144285b5eb27c32860b09d" is_pre_election: true
I20260928 20:27:10.342927 6758 raft_consensus.cc:2454] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e217fe36241e5a4df6ef68bca546c in term 0.
I20260928 20:27:10.342980 6892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68837d6fdcb4038ae61ce16fd0fb2bd" candidate_uuid: "f83e217fe36241e5a4df6ef68bca546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf84d20b28b46ed9d90060bbc716535" is_pre_election: true
I20260928 20:27:10.343163 6960 leader_election.cc:302] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d, f83e217fe36241e5a4df6ef68bca546c; no voters:
I20260928 20:27:10.343137 6892 raft_consensus.cc:2454] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83e217fe36241e5a4df6ef68bca546c in term 0.
I20260928 20:27:10.343273 7095 raft_consensus.cc:2781] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:10.343317 7095 raft_consensus.cc:482] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:10.343336 7095 raft_consensus.cc:3037] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:10.344203 7095 raft_consensus.cc:504] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.344347 7095 leader_election.cc:288] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [CANDIDATE]: Term 1 election: Requested vote from peers b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:10.344551 6892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68837d6fdcb4038ae61ce16fd0fb2bd" candidate_uuid: "f83e217fe36241e5a4df6ef68bca546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf84d20b28b46ed9d90060bbc716535"
I20260928 20:27:10.344657 6892 raft_consensus.cc:3037] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:10.344630 6758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d68837d6fdcb4038ae61ce16fd0fb2bd" candidate_uuid: "f83e217fe36241e5a4df6ef68bca546c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8f5239fd0144285b5eb27c32860b09d"
I20260928 20:27:10.344956 6758 raft_consensus.cc:3037] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:10.345850 6892 raft_consensus.cc:2454] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83e217fe36241e5a4df6ef68bca546c in term 1.
I20260928 20:27:10.345889 6758 raft_consensus.cc:2454] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83e217fe36241e5a4df6ef68bca546c in term 1.
I20260928 20:27:10.346119 6960 leader_election.cc:302] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c; no voters:
I20260928 20:27:10.346216 7095 raft_consensus.cc:2781] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:10.346411 7095 raft_consensus.cc:686] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 1 LEADER]: Becoming Leader. State: Replica: f83e217fe36241e5a4df6ef68bca546c, State: Running, Role: LEADER
I20260928 20:27:10.346539 7095 consensus_queue.cc:237] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.347419 6613 catalog_manager.cc:5495] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c reported cstate change: term changed from 0 to 1, leader changed from <none> to f83e217fe36241e5a4df6ef68bca546c (127.24.186.195). New cstate: current_term: 1 leader_uuid: "f83e217fe36241e5a4df6ef68bca546c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } health_report { overall_health: UNKNOWN } } }
I20260928 20:27:10.358636 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.356383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.24.186.195:36547)
0928 20:27:10.356449 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.356451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.356469 (+ 18us) server_negotiation.cc:407] Connection header received
0928 20:27:10.356512 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.356513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.356530 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.356586 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:10.356754 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.357226 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.358000 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.358190 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.358245 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.358364 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:10.358473 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:10.358543 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.358557 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.358575 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:27:10.361048 6892 raft_consensus.cc:1261] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Refusing update from remote peer f83e217fe36241e5a4df6ef68bca546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:10.361085 6758 raft_consensus.cc:1261] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Refusing update from remote peer f83e217fe36241e5a4df6ef68bca546c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:10.361459 7095 consensus_queue.cc:1035] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:10.361635 7099 consensus_queue.cc:1035] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:10.364671 7104 mvcc.cc:205] Tried to move back new op lower bound from 7334409135554424832 to 7334409135500447744. Current Snapshot: MvccSnapshot[applied={T|T < 7334409135554424832}]
I20260928 20:27:10.366660 6612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:10.366775 6612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:10.367599 6612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Sending DeleteTablet for 3 replicas of tablet d68837d6fdcb4038ae61ce16fd0fb2bd
I20260928 20:27:10.368021 7006 tablet_service.cc:1514] Processing DeleteTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:10 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35702
I20260928 20:27:10.368211 6738 tablet_service.cc:1514] Processing DeleteTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:10 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38606
I20260928 20:27:10.368263 7112 tablet_replica.cc:330] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: stopping tablet replica
I20260928 20:27:10.368358 7112 raft_consensus.cc:2229] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:27:10.368445 6872 tablet_service.cc:1514] Processing DeleteTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:10 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56902
I20260928 20:27:10.368489 7112 raft_consensus.cc:2258] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:10.368737 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6675
I20260928 20:27:10.369551 7112 ts_tablet_manager.cc:1905] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:10.370438 7114 tablet_replica.cc:330] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: stopping tablet replica
I20260928 20:27:10.370573 7114 raft_consensus.cc:2229] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:10.370697 7114 raft_consensus.cc:2258] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:10.371310 7114 ts_tablet_manager.cc:1905] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:10.371416 7112 ts_tablet_manager.cc:1918] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:10.371491 7112 log.cc:1196] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/d68837d6fdcb4038ae61ce16fd0fb2bd
I20260928 20:27:10.371831 7112 ts_tablet_manager.cc:1939] T d68837d6fdcb4038ae61ce16fd0fb2bd P f83e217fe36241e5a4df6ef68bca546c: Deleting consensus metadata
I20260928 20:27:10.375253 6598 catalog_manager.cc:4841] TS f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): tablet d68837d6fdcb4038ae61ce16fd0fb2bd (table test-table [id=f3ba74060a574bcc98382130cee1a9aa]) successfully deleted
W20260928 20:27:10.375710 6598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:39215 (error 108)
I20260928 20:27:10.375944 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6809
I20260928 20:27:10.376322 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.376186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39215 (local address 127.0.0.1:38622)
0928 20:27:10.376232 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.376268 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 20:27:10.376873 6598 catalog_manager.cc:4584] TS b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d68837d6fdcb4038ae61ce16fd0fb2bd: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39215: connect: Connection refused (error 111)
I20260928 20:27:10.382249 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6943
I20260928 20:27:10.382393 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.382229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.0.0.1:56908)
0928 20:27:10.382313 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.382352 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.194:39131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 20:27:10.387832 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 6581
I20260928 20:27:10.393906 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--webserver_interface=127.24.186.254
--webserver_port=44097
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:10.479275 7116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.479513 7116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.479538 7116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.481979 7116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:10.482055 7116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.482074 7116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.482086 7116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:10.482098 7116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:10.484301 7116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.484611 7116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.484876 7116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:10.494668 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.494686 7123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.494779 7116 server_base.cc:1034] running on GCE node
W20260928 20:27:10.494868 7122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.495540 7116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.496068 7116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.497264 7116 hybrid_clock.cc:648] HybridClock initialized: now 1790627230497261 us; error 43 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.500125 7116 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:10.500782 7116 webserver.cc:460] Webserver started at http://127.24.186.254:44097/ using document root <none> and password file <none>
I20260928 20:27:10.501075 7116 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.501168 7116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.502648 7116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:10.503366 7132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.503669 7116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:10.503867 7116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "a30f4b9f34fdb21fa3fc78225d1df81e"
server_key_iv: "95b3062bf7a6c11b2b9e1772922de9ee"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.504321 7116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.521085 7116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.523391 7116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.523593 7116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.529170 7116 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38383
I20260928 20:27:10.529204 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38383 every 8 connection(s)
I20260928 20:27:10.529629 7116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:10.531308 7185 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:27:10.532091 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap starting.
I20260928 20:27:10.539919 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7116
I20260928 20:27:10.541049 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:39215
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=36249
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:10.542320 7185 log.cc:824] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:10.546195 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:10.546562 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap complete.
I20260928 20:27:10.549333 7185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:10.549633 7185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Initialized, Role: FOLLOWER
I20260928 20:27:10.549856 7185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:10.549955 7185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:10.550024 7185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:10.550073 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:27:10.551182 7185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:10.551311 7185 leader_election.cc:302] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b5fa76e5f0e4cfe8d8ab653b6de079e; no voters:
I20260928 20:27:10.551530 7185 leader_election.cc:288] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:27:10.551568 7189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:27:10.551674 7189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 LEADER]: Becoming Leader. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Running, Role: LEADER
I20260928 20:27:10.551851 7185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:10.551842 7189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:10.552266 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b5fa76e5f0e4cfe8d8ab653b6de079e. Latest consensus state: current_term: 2 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:10.552421 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:10.552664 7195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:10.553458 7195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:10.553720 7190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:10.553610 7195 catalog_manager.cc:1259] Loaded cluster ID: 37df689d7c174bc4b8250b1db01acba6
I20260928 20:27:10.553933 7195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:27:10.554247 7195 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 20:27:10.555936 7190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:10.559679 7195 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 20:27:10.563333 7195 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:10.563570 7195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Loaded TSK: 0
I20260928 20:27:10.563980 7195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:27:10.631543 7188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.631769 7188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.631846 7188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.633133 7188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.633178 7188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.633206 7188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:10.634675 7188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=36249
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.634953 7188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.635216 7188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:10.641299 7213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.641276 7212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.641512 7188 server_base.cc:1034] running on GCE node
W20260928 20:27:10.641283 7215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.641835 7188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.642139 7188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.643334 7188 hybrid_clock.cc:648] HybridClock initialized: now 1790627230643318 us; error 28 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.646350 7188 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:10.646859 7188 webserver.cc:460] Webserver started at http://127.24.186.193:36249/ using document root <none> and password file <none>
I20260928 20:27:10.647107 7188 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.647202 7188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.648790 7188 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:10.649542 7222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.649778 7188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:10.649891 7188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b8f5239fd0144285b5eb27c32860b09d"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "6c8366f3bd799de0029afd925c0e1655"
server_key_iv: "be2dfe51a76d877ab50970fc611a97d7"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.650269 7188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.670550 7188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.672963 7188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.673137 7188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.673398 7188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:10.673853 7229 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:27:10.675701 7188 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:27:10.675781 7188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:27:10.675848 7188 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:27:10.676590 7188 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:27:10.676651 7188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:27:10.676707 7229 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Bootstrap starting.
I20260928 20:27:10.684473 7188 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:39215
I20260928 20:27:10.684551 7336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:39215 every 8 connection(s)
I20260928 20:27:10.684980 7188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:10.688125 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7188
I20260928 20:27:10.688437 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:39131
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=37137
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:10.688716 7229 log.cc:824] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:10.691483 7229 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:10.691789 7229 tablet_bootstrap.cc:492] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Bootstrap complete.
I20260928 20:27:10.693084 7229 ts_tablet_manager.cc:1397] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:10.694486 7229 raft_consensus.cc:348] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.694804 7229 raft_consensus.cc:729] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5239fd0144285b5eb27c32860b09d, State: Initialized, Role: FOLLOWER
I20260928 20:27:10.694988 7229 consensus_queue.cc:260] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } } peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } }
I20260928 20:27:10.695276 7229 ts_tablet_manager.cc:1428] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:27:10.701264 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.685790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:44483)
0928 20:27:10.685991 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.686046 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.686537 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.688788 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.688793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.689227 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.689511 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.689518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.690928 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.690929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.691870 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.691874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.692813 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.692814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.692985 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.693601 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.693609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.695262 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.698907 (+ 3645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.698918 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.698921 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.699243 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.699786 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.699787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.699788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.699959 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.700518 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.700525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.700691 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.700812 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.701046 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260928 20:27:10.701315 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.685930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:44483 (local address 127.24.186.254:38383)
0928 20:27:10.688185 (+ 2255us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.688197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.688251 (+ 54us) server_negotiation.cc:407] Connection header received
0928 20:27:10.688417 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.688425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.688488 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.688643 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.689655 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.690790 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.691960 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.692598 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.692710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.695408 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.695412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.695413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.695426 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.698771 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.699352 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.699357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.699357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.699496 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.700044 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.700045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.700045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.700426 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.700533 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.700894 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.701070 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2194,"thread_start_us":79,"threads_started":1}
I20260928 20:27:10.702248 7337 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:10.702368 7337 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:10.702569 7337 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:10.703296 7149 ts_manager.cc:194] Registered new tserver with Master: b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215)
I20260928 20:27:10.704373 7149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:44483
I20260928 20:27:10.705446 7337 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.193@KRBTEST.COM
I20260928 20:27:10.712863 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.704010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.24.186.193:39215)
0928 20:27:10.704702 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.704707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.704718 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:27:10.704752 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.704754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.704768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.704857 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.705275 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.706197 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.706972 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.707375 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.707467 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.709997 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.710006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.710006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.710017 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.711480 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.711930 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.711934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.711934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.712005 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.712216 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.712217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.712217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.712376 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.712468 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.712670 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.712766 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":84,"threads_started":1}
I20260928 20:27:10.712863 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.703953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39215 (local address 127.0.0.1:38628)
0928 20:27:10.704230 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.704265 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.704333 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.704859 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.704864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.705141 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.705186 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.705188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.706277 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.706278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.706875 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.706877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.707471 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.707472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.707561 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.708166 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.708172 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.709884 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.711589 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.711596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.711598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.711824 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.712080 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.712081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.712081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.712144 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.712459 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.712473 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.712526 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.712622 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.712756 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":116,"threads_started":1}
I20260928 20:27:10.713289 7271 tablet_service.cc:1514] Processing DeleteTablet for tablet d68837d6fdcb4038ae61ce16fd0fb2bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:10 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38628
I20260928 20:27:10.713840 7346 tablet_replica.cc:330] stopping tablet replica
I20260928 20:27:10.713977 7346 raft_consensus.cc:2229] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:10.714056 7346 raft_consensus.cc:2258] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:10.714380 7346 ts_tablet_manager.cc:1905] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:10.716777 7346 ts_tablet_manager.cc:1918] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:10.716871 7346 log.cc:1196] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/d68837d6fdcb4038ae61ce16fd0fb2bd
I20260928 20:27:10.717118 7346 ts_tablet_manager.cc:1939] T d68837d6fdcb4038ae61ce16fd0fb2bd P b8f5239fd0144285b5eb27c32860b09d: Deleting consensus metadata
I20260928 20:27:10.717963 7134 catalog_manager.cc:4841] TS b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215): tablet d68837d6fdcb4038ae61ce16fd0fb2bd (table test-table [id=f3ba74060a574bcc98382130cee1a9aa]) successfully deleted
W20260928 20:27:10.789563 7342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.789796 7342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.789826 7342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.791153 7342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.791203 7342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.791232 7342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:10.792917 7342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:39131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=37137
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.793193 7342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.793442 7342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:10.798945 7354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.799022 7351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.799130 7342 server_base.cc:1034] running on GCE node
W20260928 20:27:10.798954 7352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.799437 7342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.799739 7342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.800935 7342 hybrid_clock.cc:648] HybridClock initialized: now 1790627230800909 us; error 45 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.805300 7342 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:10.805774 7342 webserver.cc:460] Webserver started at http://127.24.186.194:37137/ using document root <none> and password file <none>
I20260928 20:27:10.806031 7342 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.806098 7342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.807684 7342 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:10.808490 7361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.808674 7342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:10.808795 7342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "caf84d20b28b46ed9d90060bbc716535"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "93963f72723539e213cf18f608a1619a"
server_key_iv: "0b6fcdbdf178f9b718c67c2d160bbd71"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.809170 7342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.832266 7342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.834378 7342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.834508 7342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.834705 7342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:10.835186 7368 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 20:27:10.837141 7368 ts_tablet_manager.cc:1852] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 20:27:10.837421 7368 ts_tablet_manager.cc:1905] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:10.839684 7368 ts_tablet_manager.cc:1918] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:10.840020 7368 log.cc:1196] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/d68837d6fdcb4038ae61ce16fd0fb2bd
I20260928 20:27:10.840343 7368 ts_tablet_manager.cc:1939] T d68837d6fdcb4038ae61ce16fd0fb2bd P caf84d20b28b46ed9d90060bbc716535: Deleting consensus metadata
I20260928 20:27:10.840618 7342 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 20:27:10.840706 7342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 20:27:10.840761 7342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:10.840795 7342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.847240 7342 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:39131
I20260928 20:27:10.847299 7475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:39131 every 8 connection(s)
I20260928 20:27:10.847617 7342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:27:10.850677 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7342
I20260928 20:27:10.850971 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:36547
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=46047
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:10.858587 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.848390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:55461)
0928 20:27:10.848555 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.848591 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.849105 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.849362 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.849366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.849614 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.849805 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.849808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.850501 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.850502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.851579 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.851582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.852054 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.852056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.852224 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.852852 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.852863 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.854806 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.856975 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.856983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.856985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.857230 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.857486 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.857487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.857487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.857626 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.858025 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.858032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.858175 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.858268 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.858407 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 20:27:10.858681 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.848385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:55461 (local address 127.24.186.254:38383)
0928 20:27:10.848455 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.848457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.848678 (+ 221us) server_negotiation.cc:407] Connection header received
0928 20:27:10.849216 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.849218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.849232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.849291 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.849902 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.850401 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.851684 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.851957 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.852044 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.854920 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.854922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.854923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.854932 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.856875 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.857315 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.857317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.857317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.857394 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.857709 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.857710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.857711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.857882 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.857962 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.858319 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.858443 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:27:10.859380 7476 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:10.859506 7476 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:10.859697 7476 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:10.860205 7149 ts_manager.cc:194] Registered new tserver with Master: caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:10.860769 7149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:55461
W20260928 20:27:10.940905 7479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:10.941148 7479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:10.941176 7479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:10.943758 7479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:10.943886 7479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:10.943925 7479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:10.946005 7479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:36547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=46047
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:10.946224 7479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:10.946429 7479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:10.951442 7485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.951407 7487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:10.951393 7484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:10.951831 7479 server_base.cc:1034] running on GCE node
I20260928 20:27:10.952021 7479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:10.952224 7479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:10.953471 7479 hybrid_clock.cc:648] HybridClock initialized: now 1790627230953442 us; error 29 us; skew 500 ppm
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:10.956035 7479 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:10.956547 7479 webserver.cc:460] Webserver started at http://127.24.186.195:46047/ using document root <none> and password file <none>
I20260928 20:27:10.956796 7479 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:10.956878 7479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:10.958236 7479 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:10.958808 7494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.958989 7479 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:27:10.959085 7479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "f83e217fe36241e5a4df6ef68bca546c"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "9afa77d052df54e1d29a27a9380f19af"
server_key_iv: "11feb1702fa220976b32e06c7e57f5e6"
server_key_version: "encryptionkey@0"
I20260928 20:27:10.959452 7479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:10.969220 7479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:10.971598 7479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:10.971783 7479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:10.972108 7479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:10.972463 7479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:10.972519 7479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.972563 7479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:10.972592 7479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:10.978717 7479 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:36547
I20260928 20:27:10.978757 7607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:36547 every 8 connection(s)
I20260928 20:27:10.979132 7479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:27:10 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627230, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:10.987946 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7479
I20260928 20:27:10.989274 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.980062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:46407 (local address 127.24.186.254:38383)
0928 20:27:10.980110 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.980113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.980319 (+ 206us) server_negotiation.cc:407] Connection header received
0928 20:27:10.980862 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.980863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.980877 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.980937 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.981960 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.982522 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.983530 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.983733 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.983861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.986017 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.986019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.986019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.986026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.987386 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.988177 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.988179 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.988185 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.988253 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.988553 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.988554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.988554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.988686 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.988768 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.989065 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.989169 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:27:10.989511 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.980032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:46407)
0928 20:27:10.980198 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:10.980242 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:10.980748 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:10.981022 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:10.981026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:10.981620 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:10.981835 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.981839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.982625 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.982626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.983431 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:10.983433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.983844 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.983845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:10.983988 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.984515 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:10.984523 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:10.985930 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:10.987485 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.987491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.987493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.988062 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.988355 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:10.988356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:10.988356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:10.988480 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:10.988769 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:10.988776 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:10.988926 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 20:27:10.989042 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:27:10.989245 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 20:27:10.990083 7608 heartbeater.cc:347] Connected to a master server at 127.24.186.254:38383
I20260928 20:27:10.990218 7608 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:10.990434 7608 heartbeater.cc:511] Master 127.24.186.254:38383 requested a full tablet report, sending...
I20260928 20:27:10.991104 7149 ts_manager.cc:194] Registered new tserver with Master: f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547)
I20260928 20:27:10.991482 7149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:46407
I20260928 20:27:10.995095 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.988381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.24.186.254:38383)
0928 20:27:10.988507 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.988509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.988548 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:27:10.988631 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.988632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.988644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.988719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:10.989312 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.990084 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.991026 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.991318 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.991392 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.992673 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:10.992674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:10.992675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:10.992682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:10.993920 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.994287 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.994288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.994289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.994346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:10.994550 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:10.994551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:10.994552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:10.994667 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:10.994848 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.994883 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.994966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 20:27:10.995751 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:27:10.999081 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:10.996157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.24.186.254:38383)
0928 20:27:10.996223 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:10.996225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:10.996348 (+ 123us) server_negotiation.cc:407] Connection header received
0928 20:27:10.996395 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:10.996396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:10.996422 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:10.996470 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:10.996555 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.997115 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.998235 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:10.998438 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:10.998505 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:10.998731 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:10.998837 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:10.998916 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:10.998999 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 20:27:10.999023 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:27:10.999398 7149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:27:11.001092 7149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:27:11.003567 7271 tablet_service.cc:1467] Processing CreateTablet for tablet b14af2afba5f472dbbe34adbe184d4a3 (DEFAULT_TABLE table=test-table [id=46193ab69ea8452d97b963bf486c7246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:11.003785 7271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14af2afba5f472dbbe34adbe184d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:11.005208 7229 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Bootstrap starting.
I20260928 20:27:11.005597 7229 tablet_bootstrap.cc:654] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:11.006167 7229 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: No bootstrap required, opened a new log
I20260928 20:27:11.006229 7229 ts_tablet_manager.cc:1397] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:11.006693 7229 raft_consensus.cc:348] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.006826 7229 raft_consensus.cc:374] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:11.006969 7229 raft_consensus.cc:729] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5239fd0144285b5eb27c32860b09d, State: Initialized, Role: FOLLOWER
I20260928 20:27:11.007062 7229 consensus_queue.cc:260] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.007174 7229 ts_tablet_manager.cc:1428] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:11.008312 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.003131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.0.0.1:56914)
0928 20:27:11.003364 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.003375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:11.003420 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:11.004119 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:11.004121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:11.004123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:11.004173 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.004175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.005048 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.005050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.006329 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.006332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.008040 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.008041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.008107 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.008126 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 20:27:11.008181 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 20:27:11.008230 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
I20260928 20:27:11.008323 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.003256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.24.186.194:39131)
0928 20:27:11.003936 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:11.003940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:11.003960 (+ 20us) server_negotiation.cc:407] Connection header received
0928 20:27:11.003992 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:11.003994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:11.004007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:11.004099 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:11.004244 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.004941 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.006412 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.007961 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.008041 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.008145 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:11.008200 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 20:27:11.008242 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":74,"threads_started":1}
I20260928 20:27:11.008973 7410 tablet_service.cc:1467] Processing CreateTablet for tablet b14af2afba5f472dbbe34adbe184d4a3 (DEFAULT_TABLE table=test-table [id=46193ab69ea8452d97b963bf486c7246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:11.009269 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.003035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36547 (local address 127.0.0.1:35720)
0928 20:27:11.003093 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.003106 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:11.003153 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:11.005651 (+ 2498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:11.005652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:11.005654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:11.005692 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.005694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.006820 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.006821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.007852 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.007854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.009021 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.009023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.009091 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.009096 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:27:11.009148 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:27:11.009203 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 20:27:11.009328 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.003151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.24.186.195:36547)
0928 20:27:11.005471 (+ 2320us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:11.005476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:11.005486 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:11.005528 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:11.005530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:11.005547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:11.005756 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:11.005832 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.006709 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.007923 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.008916 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.009012 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.009064 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:11.009186 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 20:27:11.009232 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2271,"thread_start_us":125,"threads_started":1}
I20260928 20:27:11.009342 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14af2afba5f472dbbe34adbe184d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:11.010159 7542 tablet_service.cc:1467] Processing CreateTablet for tablet b14af2afba5f472dbbe34adbe184d4a3 (DEFAULT_TABLE table=test-table [id=46193ab69ea8452d97b963bf486c7246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:11.010504 7542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b14af2afba5f472dbbe34adbe184d4a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:11.011914 7368 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Bootstrap starting.
I20260928 20:27:11.012764 7368 tablet_bootstrap.cc:654] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:11.013119 7368 log.cc:824] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:11.013851 7619 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Bootstrap starting.
I20260928 20:27:11.013916 7368 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: No bootstrap required, opened a new log
I20260928 20:27:11.014045 7368 ts_tablet_manager.cc:1397] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:11.014613 7619 tablet_bootstrap.cc:654] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:11.015193 7619 log.cc:824] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:11.016113 7368 raft_consensus.cc:348] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.016284 7368 raft_consensus.cc:374] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:11.016325 7368 raft_consensus.cc:729] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf84d20b28b46ed9d90060bbc716535, State: Initialized, Role: FOLLOWER
I20260928 20:27:11.016456 7368 consensus_queue.cc:260] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.016564 7343 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:11.016705 7343 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.016839 7619 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: No bootstrap required, opened a new log
I20260928 20:27:11.016927 7619 ts_tablet_manager.cc:1397] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:27:11.017082 7343 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:11.018172 7368 ts_tablet_manager.cc:1428] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 20:27:11.018316 7476 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
I20260928 20:27:11.019491 7619 raft_consensus.cc:348] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.019691 7619 raft_consensus.cc:374] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:11.019735 7619 raft_consensus.cc:729] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83e217fe36241e5a4df6ef68bca546c, State: Initialized, Role: FOLLOWER
I20260928 20:27:11.019936 7619 consensus_queue.cc:260] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.020255 7608 heartbeater.cc:503] Master 127.24.186.254:38383 was elected leader, sending a full tablet report...
I20260928 20:27:11.020249 7619 ts_tablet_manager.cc:1428] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:27:11.021090 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.017322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52193 (local address 127.24.186.195:36547)
0928 20:27:11.017364 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:11.017367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:11.017564 (+ 197us) server_negotiation.cc:407] Connection header received
0928 20:27:11.017656 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:11.017657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:11.017667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:11.017713 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:11.017884 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.018600 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.020080 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.020647 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.020695 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.020703 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:11.020982 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 20:27:11.021024 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:27:11.021117 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.017353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.24.186.193:54251)
0928 20:27:11.017496 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.017507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:11.017720 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:11.018054 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:11.018055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:11.018057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:11.018108 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.018109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.019107 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.019108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.020146 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.020148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.020881 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.020882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.020956 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.020959 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:27:11.020987 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 20:27:11.021030 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260928 20:27:11.021297 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.017260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36547 (local address 127.24.186.193:52193)
0928 20:27:11.017495 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.017505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:11.017589 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:11.017724 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:11.017726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:11.017727 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:11.017796 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.017798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.018690 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.018691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.019983 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:11.019986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.020781 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.020782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:11.020874 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.020897 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 20:27:11.021177 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 20:27:11.021228 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260928 20:27:11.021405 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.017888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:54251 (local address 127.24.186.194:39131)
0928 20:27:11.017935 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:11.017937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:11.017945 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:11.017965 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:11.017965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:11.017975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:11.018162 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:11.018194 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.019009 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.020232 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.020795 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.021263 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.021272 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:11.021292 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 20:27:11.021335 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:27:11.021629 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14af2afba5f472dbbe34adbe184d4a3" candidate_uuid: "b8f5239fd0144285b5eb27c32860b09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83e217fe36241e5a4df6ef68bca546c" is_pre_election: true
I20260928 20:27:11.021689 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14af2afba5f472dbbe34adbe184d4a3" candidate_uuid: "b8f5239fd0144285b5eb27c32860b09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf84d20b28b46ed9d90060bbc716535" is_pre_election: true
I20260928 20:27:11.021775 7562 raft_consensus.cc:2454] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5239fd0144285b5eb27c32860b09d in term 0.
I20260928 20:27:11.021871 7430 raft_consensus.cc:2454] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8f5239fd0144285b5eb27c32860b09d in term 0.
I20260928 20:27:11.022013 7224 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d, f83e217fe36241e5a4df6ef68bca546c; no voters:
I20260928 20:27:11.022202 7343 raft_consensus.cc:2781] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:11.022259 7343 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:11.022276 7343 raft_consensus.cc:3037] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:11.022936 7343 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.023074 7343 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 1 election: Requested vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
I20260928 20:27:11.023245 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14af2afba5f472dbbe34adbe184d4a3" candidate_uuid: "b8f5239fd0144285b5eb27c32860b09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf84d20b28b46ed9d90060bbc716535"
I20260928 20:27:11.023285 7562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b14af2afba5f472dbbe34adbe184d4a3" candidate_uuid: "b8f5239fd0144285b5eb27c32860b09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83e217fe36241e5a4df6ef68bca546c"
I20260928 20:27:11.023352 7430 raft_consensus.cc:3037] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:11.023353 7562 raft_consensus.cc:3037] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:11.024201 7562 raft_consensus.cc:2454] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5239fd0144285b5eb27c32860b09d in term 1.
I20260928 20:27:11.024259 7430 raft_consensus.cc:2454] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8f5239fd0144285b5eb27c32860b09d in term 1.
I20260928 20:27:11.024359 7224 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d, f83e217fe36241e5a4df6ef68bca546c; no voters:
I20260928 20:27:11.024459 7343 raft_consensus.cc:2781] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:11.024638 7343 raft_consensus.cc:686] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 LEADER]: Becoming Leader. State: Replica: b8f5239fd0144285b5eb27c32860b09d, State: Running, Role: LEADER
I20260928 20:27:11.024709 7343 consensus_queue.cc:237] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.025468 7149 catalog_manager.cc:5495] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d reported cstate change: term changed from 0 to 1, leader changed from <none> to b8f5239fd0144285b5eb27c32860b09d (127.24.186.193). New cstate: current_term: 1 leader_uuid: "b8f5239fd0144285b5eb27c32860b09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } health_report { overall_health: HEALTHY } } }
I20260928 20:27:11.031981 7345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.029765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.24.186.193:39215)
0928 20:27:11.029824 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:11.029825 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:11.029834 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:27:11.029896 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:11.029897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:11.029913 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:11.029975 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:11.030072 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.030563 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.031293 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:11.031490 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:11.031538 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:11.031656 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:11.031748 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:11.031792 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:11.031900 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 20:27:11.031919 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:27:11.033478 7430 raft_consensus.cc:1261] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Refusing update from remote peer b8f5239fd0144285b5eb27c32860b09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:11.033471 7562 raft_consensus.cc:1261] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Refusing update from remote peer b8f5239fd0144285b5eb27c32860b09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:11.033811 7343 consensus_queue.cc:1035] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:11.033973 7624 consensus_queue.cc:1035] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:11.036298 7627 mvcc.cc:205] Tried to move back new op lower bound from 7334409138310443008 to 7334409138277822464. Current Snapshot: MvccSnapshot[applied={T|T < 7334409138310443008}]
I20260928 20:27:11.038633 7148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:11.038739 7148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:11.039516 7148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Sending DeleteTablet for 3 replicas of tablet b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:11.039875 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7188
I20260928 20:27:11.039928 7410 tablet_service.cc:1514] Processing DeleteTablet for tablet b14af2afba5f472dbbe34adbe184d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:11 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56914
I20260928 20:27:11.039938 7542 tablet_service.cc:1514] Processing DeleteTablet for tablet b14af2afba5f472dbbe34adbe184d4a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:11 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35720
I20260928 20:27:11.040690 7637 tablet_replica.cc:330] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: stopping tablet replica
I20260928 20:27:11.040817 7637 raft_consensus.cc:2229] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:11.040968 7637 raft_consensus.cc:2258] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:11.041174 7636 tablet_replica.cc:330] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: stopping tablet replica
I20260928 20:27:11.041280 7636 raft_consensus.cc:2229] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:11.041381 7636 raft_consensus.cc:2258] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:11.041790 7637 ts_tablet_manager.cc:1905] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:11.043867 7637 ts_tablet_manager.cc:1918] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:11.043933 7637 log.cc:1196] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:11.044166 7637 ts_tablet_manager.cc:1939] T b14af2afba5f472dbbe34adbe184d4a3 P caf84d20b28b46ed9d90060bbc716535: Deleting consensus metadata
I20260928 20:27:11.044241 7636 ts_tablet_manager.cc:1905] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:11.044646 7134 catalog_manager.cc:4841] TS caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): tablet b14af2afba5f472dbbe34adbe184d4a3 (table test-table [id=46193ab69ea8452d97b963bf486c7246]) successfully deleted
I20260928 20:27:11.045837 7636 ts_tablet_manager.cc:1918] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:11.045945 7636 log.cc:1196] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:11.046216 7636 ts_tablet_manager.cc:1939] T b14af2afba5f472dbbe34adbe184d4a3 P f83e217fe36241e5a4df6ef68bca546c: Deleting consensus metadata
I20260928 20:27:11.046648 7134 catalog_manager.cc:4841] TS f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): tablet b14af2afba5f472dbbe34adbe184d4a3 (table test-table [id=46193ab69ea8452d97b963bf486c7246]) successfully deleted
W20260928 20:27:11.047102 7134 connection.cc:535] client connection to 127.24.186.193:39215 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 20:27:11.047194 7134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:27:11.047449 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7342
I20260928 20:27:11.048172 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.047535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39215 (local address 127.0.0.1:38644)
0928 20:27:11.048087 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.048133 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528}
W20260928 20:27:11.048439 7134 catalog_manager.cc:4584] TS b8f5239fd0144285b5eb27c32860b09d (127.24.186.193:39215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b14af2afba5f472dbbe34adbe184d4a3: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39215: connect: Connection refused (error 111)
I20260928 20:27:11.053239 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7479
I20260928 20:27:11.059274 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7116
I20260928 20:27:11.065389 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--webserver_interface=127.24.186.254
--webserver_port=44097
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:11.154461 7639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:11.154692 7639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:11.154717 7639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:11.156101 7639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:11.156162 7639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:11.156176 7639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:11.156188 7639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:11.156201 7639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:11.157848 7639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:11.158138 7639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:11.158389 7639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:11.163579 7644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.163666 7645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.163607 7647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:11.163897 7639 server_base.cc:1034] running on GCE node
I20260928 20:27:11.164201 7639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:11.164513 7639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:11.165709 7639 hybrid_clock.cc:648] HybridClock initialized: now 1790627231165680 us; error 43 us; skew 500 ppm
Sep 28 20:27:11 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:11.168479 7639 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:11.168967 7639 webserver.cc:460] Webserver started at http://127.24.186.254:44097/ using document root <none> and password file <none>
I20260928 20:27:11.169229 7639 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:11.169279 7639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:11.171032 7639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:11.172089 7654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.172291 7639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:11.172386 7639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "a30f4b9f34fdb21fa3fc78225d1df81e"
server_key_iv: "95b3062bf7a6c11b2b9e1772922de9ee"
server_key_version: "encryptionkey@0"
I20260928 20:27:11.172772 7639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:11.184487 7639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:11.188079 7639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:11.188249 7639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:11.192793 7639 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38383
I20260928 20:27:11.192828 7706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38383 every 8 connection(s)
I20260928 20:27:11.193269 7639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:11.194903 7707 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:27:11.195574 7707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap starting.
I20260928 20:27:11.201200 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7639
I20260928 20:27:11.201515 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:39215
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=36249
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:11.206248 7707 log.cc:824] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:11.209141 7707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:11.209506 7707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e: Bootstrap complete.
I20260928 20:27:11.211918 7707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:11.212206 7707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Initialized, Role: FOLLOWER
I20260928 20:27:11.212389 7707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:11.212471 7707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:11.212524 7707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:11.212576 7707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 2 FOLLOWER]: Advancing to term 3
I20260928 20:27:11.213621 7707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:11.213747 7707 leader_election.cc:302] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b5fa76e5f0e4cfe8d8ab653b6de079e; no voters:
I20260928 20:27:11.213972 7707 leader_election.cc:288] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 20:27:11.214008 7711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 20:27:11.214126 7711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [term 3 LEADER]: Becoming Leader. State: Replica: 9b5fa76e5f0e4cfe8d8ab653b6de079e, State: Running, Role: LEADER
I20260928 20:27:11.214258 7707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:11.214237 7711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } }
I20260928 20:27:11.214574 7711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b5fa76e5f0e4cfe8d8ab653b6de079e. Latest consensus state: current_term: 3 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:11.214641 7711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:11.214784 7712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b5fa76e5f0e4cfe8d8ab653b6de079e" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38383 } } }
I20260928 20:27:11.214854 7715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:11.214883 7712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b5fa76e5f0e4cfe8d8ab653b6de079e [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:11.215844 7715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:11.216004 7715 catalog_manager.cc:1259] Loaded cluster ID: 37df689d7c174bc4b8250b1db01acba6
I20260928 20:27:11.216051 7715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 20:27:11.216409 7715 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc8c85bf9ad google::LogMessage::Fail() at ??:0
@ 0x7fc8c85c288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc8c85bf4de google::LogMessage::Flush() at ??:0
@ 0x7fc8c85c0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc8d04e40f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fc8d04ed8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc8ca48d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc8ca482954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc8cb2e66db start_thread at ??:0
@ 0x7fc8c70bd61f clone at ??:0
W20260928 20:27:11.300500 7709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:11.300710 7709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:11.300735 7709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:11.301944 7709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:11.301993 7709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:11.302022 7709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:11.303586 7709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=36249
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:11.303879 7709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:11.304138 7709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:11.309147 7744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.309161 7743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.309161 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:11.309408 7709 server_base.cc:1034] running on GCE node
I20260928 20:27:11.309881 7709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:11.310187 7709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:11.311388 7709 hybrid_clock.cc:648] HybridClock initialized: now 1790627231311374 us; error 34 us; skew 500 ppm
Sep 28 20:27:11 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:11.314472 7709 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:11.315012 7709 webserver.cc:460] Webserver started at http://127.24.186.193:36249/ using document root <none> and password file <none>
I20260928 20:27:11.315254 7709 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:11.315321 7709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:11.316867 7709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:11.318113 7753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.318660 7709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:11.318954 7709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b8f5239fd0144285b5eb27c32860b09d"
format_stamp: "Formatted at 2026-09-28 20:27:09 on dist-test-slave-k477"
server_key: "6c8366f3bd799de0029afd925c0e1655"
server_key_iv: "be2dfe51a76d877ab50970fc611a97d7"
server_key_version: "encryptionkey@0"
I20260928 20:27:11.319370 7709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:11.336081 7709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:11.338511 7709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:11.338804 7709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:11.339083 7709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:11.339545 7760 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:27:11.341703 7709 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:27:11.341795 7709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:27:11.341864 7709 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:27:11.342808 7709 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:27:11.342931 7760 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Bootstrap starting.
I20260928 20:27:11.342944 7709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:27:11.350944 7709 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:39215
I20260928 20:27:11.351049 7867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:39215 every 8 connection(s)
I20260928 20:27:11.351399 7709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:11.355314 7760 log.cc:824] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:11.358196 7760 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:11.358536 7760 tablet_bootstrap.cc:492] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Bootstrap complete.
I20260928 20:27:11.358536 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7709
I20260928 20:27:11.358803 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:39131
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=37137
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:11.359845 7760 ts_tablet_manager.cc:1397] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260928 20:27:11.362074 7760 raft_consensus.cc:348] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.362474 7760 raft_consensus.cc:729] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8f5239fd0144285b5eb27c32860b09d, State: Initialized, Role: FOLLOWER
I20260928 20:27:11.362882 7760 consensus_queue.cc:260] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:11.363229 7760 ts_tablet_manager.cc:1428] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260928 20:27:11.429843 7870 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:27:11.352215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:37479)
0928 20:27:11.352434 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.352475 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:11.352952 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:11.429628 (+ 76676us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":102,"threads_started":1}
W20260928 20:27:11.430130 7868 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:38383 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:38383: Client connection negotiation failed: client connection to 127.24.186.254:38383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:27:11.430493 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.430363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:42177)
0928 20:27:11.430423 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.430474 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 20:27:11.430845 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.430751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:54585)
0928 20:27:11.430800 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.430824 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:27:11.430994 7868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:27:11.451423 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:11.451646 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:11.451673 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:11.452862 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:11.452911 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:11.452941 7872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:11.454370 7872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:39131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=37137
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:11.454581 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:11.454767 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:11.460659 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.460753 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.460695 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:11.460913 7872 server_base.cc:1034] running on GCE node
I20260928 20:27:11.461134 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:11.461360 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:11.462507 7872 hybrid_clock.cc:648] HybridClock initialized: now 1790627231462492 us; error 26 us; skew 500 ppm
Sep 28 20:27:11 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:11.465260 7872 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:11.466446 7872 webserver.cc:460] Webserver started at http://127.24.186.194:37137/ using document root <none> and password file <none>
I20260928 20:27:11.466869 7872 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:11.466948 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:11.468549 7872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:27:11.469272 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.469544 7872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:11.469627 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "caf84d20b28b46ed9d90060bbc716535"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "93963f72723539e213cf18f608a1619a"
server_key_iv: "0b6fcdbdf178f9b718c67c2d160bbd71"
server_key_version: "encryptionkey@0"
I20260928 20:27:11.470010 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:11.493233 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:11.495237 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:11.495404 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:11.495641 7872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:11.495990 7872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:11.496029 7872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.496076 7872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:11.496104 7872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.502588 7872 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:39131
I20260928 20:27:11.502635 8001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:39131 every 8 connection(s)
I20260928 20:27:11.502981 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:27:11.504140 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.503672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:46105)
0928 20:27:11.503906 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.503997 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20260928 20:27:11.504549 8002 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:38383 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:38383: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
I20260928 20:27:11.504873 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.504773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:33075)
0928 20:27:11.504820 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.504851 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:27:11.505182 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.505106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:42869)
0928 20:27:11.505145 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.505166 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 20:27:11.505330 8002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:27:11.506744 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 7872
I20260928 20:27:11.507043 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:36547
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:38383
--webserver_port=46047
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:11.590500 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:11.590762 8005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:11.590801 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:11.591993 8005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:11.592068 8005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:11.592116 8005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:11.593627 8005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:36547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=46047
--tserver_master_addrs=127.24.186.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:11.593909 8005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:11.594154 8005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:11.599195 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:11.599198 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:11.599426 8005 server_base.cc:1034] running on GCE node
W20260928 20:27:11.599200 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:11.599720 8005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:11.600006 8005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:11.601176 8005 hybrid_clock.cc:648] HybridClock initialized: now 1790627231601170 us; error 38 us; skew 500 ppm
Sep 28 20:27:11 dist-test-slave-k477 krb5kdc[6568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:11.603845 8005 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:11.604324 8005 webserver.cc:460] Webserver started at http://127.24.186.195:46047/ using document root <none> and password file <none>
I20260928 20:27:11.604583 8005 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:11.604656 8005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:11.606071 8005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:11.606786 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.607118 8005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:11.607226 8005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "f83e217fe36241e5a4df6ef68bca546c"
format_stamp: "Formatted at 2026-09-28 20:27:10 on dist-test-slave-k477"
server_key: "9afa77d052df54e1d29a27a9380f19af"
server_key_iv: "11feb1702fa220976b32e06c7e57f5e6"
server_key_version: "encryptionkey@0"
I20260928 20:27:11.607606 8005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:11.617887 8005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:11.620172 8005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:11.620361 8005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:11.620607 8005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:11.620934 8005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:11.620990 8005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.621035 8005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:11.621068 8005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:11.628016 8005 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:36547
I20260928 20:27:11.628100 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:36547 every 8 connection(s)
I20260928 20:27:11.628471 8005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:27:11.629659 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.629191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:47643)
0928 20:27:11.629402 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.629488 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":85,"threads_started":1}
W20260928 20:27:11.630079 8134 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:38383 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:38383: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
I20260928 20:27:11.630410 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.630266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:43279)
0928 20:27:11.630358 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.630389 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 20:27:11.630788 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:11.630687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:39283)
0928 20:27:11.630737 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:11.630762 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:27:11.630923 8134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:27:11.632556 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8005
I20260928 20:27:12.432349 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:12.431526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:34161)
0928 20:27:12.432163 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:12.432280 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":323,"threads_started":1}
I20260928 20:27:12.505882 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:12.505669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:40299)
0928 20:27:12.505791 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:12.505832 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260928 20:27:12.631659 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:12.631318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:37919)
0928 20:27:12.631465 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:12.631506 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260928 20:27:13.035380 8141 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:13.035526 8141 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:13.035883 8141 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
Sep 28 20:27:13 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.195@KRBTEST.COM
Sep 28 20:27:13 dist-test-slave-k477 krb5kdc[6568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627231, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.194@KRBTEST.COM
I20260928 20:27:13.046965 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.036107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:36547 (local address 127.24.186.193:41773)
0928 20:27:13.036384 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:13.036399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:13.036464 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:13.037266 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:13.037269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:13.037679 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:13.037860 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:13.037866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.038985 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.038986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:13.039904 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:13.039907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.040570 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.040571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:13.040666 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:13.041445 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:13.041453 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:13.043052 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:13.045318 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:13.045326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:13.045329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:13.045588 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:13.045869 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:13.045869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:13.045870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:13.045977 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:13.046386 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:13.046392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:13.046532 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 20:27:13.046662 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 20:27:13.046864 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":10,"thread_start_us":71,"threads_started":1}
I20260928 20:27:13.046977 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.036224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:41773 (local address 127.24.186.195:36547)
0928 20:27:13.036473 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:13.036486 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:13.036529 (+ 43us) server_negotiation.cc:407] Connection header received
0928 20:27:13.037021 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:13.037024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:13.037069 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:13.037191 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:13.038020 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.038870 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.039996 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.040452 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.040591 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:13.043185 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:13.043189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:13.043190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:13.043207 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:13.045196 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:13.045682 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:13.045685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:13.045686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:13.045788 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:13.046052 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:13.046053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:13.046053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:13.046311 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:13.046407 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:13.046717 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 20:27:13.046852 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":53,"threads_started":1}
I20260928 20:27:13.047008 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.036421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52953 (local address 127.24.186.194:39131)
0928 20:27:13.036686 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:13.036702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:13.036722 (+ 20us) server_negotiation.cc:407] Connection header received
0928 20:27:13.037128 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:13.037130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:13.037198 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:13.037363 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:13.037990 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.039070 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.040041 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.040448 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.040569 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:13.043721 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:13.043726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:13.043726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:13.043739 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:13.045462 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:13.045836 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:13.045839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:13.045839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:13.045929 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:13.046153 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:13.046155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:13.046155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:13.046396 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:13.046492 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:13.046720 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 20:27:13.046855 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":67,"threads_started":1}
I20260928 20:27:13.047092 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.036288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:39131 (local address 127.24.186.193:52953)
0928 20:27:13.036401 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:13.036413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:13.036493 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:13.037354 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:13.037355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:13.037677 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:13.037866 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:13.037868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.039176 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.039177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:13.039920 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:13.039923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:13.040544 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:13.040545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:13.040666 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:13.041504 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:13.041508 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:13.043596 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:13.045577 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:13.045578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:13.045578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:13.045752 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:13.046018 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:13.046018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:13.046019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:13.046079 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:13.046474 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:13.046475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:13.046533 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 20:27:13.046662 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:27:13.046949 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":139,"thread_start_us":55,"threads_started":1}
W20260928 20:27:13.048465 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
W20260928 20:27:13.048552 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:13.048570 7755 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d; no voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c
I20260928 20:27:13.048678 8141 raft_consensus.cc:2726] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:27:13.433168 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.433007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:49543)
0928 20:27:13.433081 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:13.433127 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 20:27:13.506680 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.506422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:49955)
0928 20:27:13.506575 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:13.506630 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260928 20:27:13.632721 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:13.632376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:60539)
0928 20:27:13.632590 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:13.632666 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
I20260928 20:27:14.433986 8148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:14.433740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:53977)
0928 20:27:14.433888 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:14.433931 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260928 20:27:14.507553 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:14.507306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:41589)
0928 20:27:14.507448 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:14.507492 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":63,"threads_started":1}
I20260928 20:27:14.633596 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:14.633337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:44413)
0928 20:27:14.633506 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:14.633543 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260928 20:27:14.847875 8151 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:14.847998 8151 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:14.848150 8151 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
W20260928 20:27:14.848613 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
W20260928 20:27:14.848714 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:14.848794 7755 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d; no voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c
I20260928 20:27:14.848884 8151 raft_consensus.cc:2726] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:27:15.434923 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:15.434626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:42047)
0928 20:27:15.434812 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:15.434854 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260928 20:27:15.508817 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:15.508302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:41369)
0928 20:27:15.508528 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:15.508701 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
I20260928 20:27:15.634418 8154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:15.634170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:40273)
0928 20:27:15.634330 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:15.634373 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260928 20:27:16.435784 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:16.435555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:38729)
0928 20:27:16.435690 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:16.435737 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 20:27:16.509800 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:16.509479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:60129)
0928 20:27:16.509657 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:16.509738 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260928 20:27:16.635221 8157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:16.634963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:49401)
0928 20:27:16.635104 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:16.635149 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260928 20:27:16.642151 8158 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:16.642262 8158 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:16.642473 8158 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
W20260928 20:27:16.642926 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
W20260928 20:27:16.643028 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:16.643062 7755 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d; no voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c
I20260928 20:27:16.643165 8158 raft_consensus.cc:2726] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:27:17.437013 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:17.436694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:45015)
0928 20:27:17.436905 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:17.436962 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":105,"threads_started":1}
I20260928 20:27:17.510798 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:17.510359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:49971)
0928 20:27:17.510570 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:17.510656 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260928 20:27:17.636078 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:17.635861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:45409)
0928 20:27:17.635989 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:17.636030 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260928 20:27:18.437898 8162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:18.437645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:37857)
0928 20:27:18.437802 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:18.437850 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260928 20:27:18.511898 8163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:18.511507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:47693)
0928 20:27:18.511739 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:18.511783 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":141,"threads_started":1}
I20260928 20:27:18.637164 8164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:18.636839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:52817)
0928 20:27:18.637056 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:18.637114 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
I20260928 20:27:19.438776 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:19.438511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:41739)
0928 20:27:19.438682 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:19.438728 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260928 20:27:19.512681 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:19.512433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:60593)
0928 20:27:19.512587 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:19.512633 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":60,"threads_started":1}
I20260928 20:27:19.638300 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:19.637985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:55001)
0928 20:27:19.638190 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:19.638248 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
I20260928 20:27:19.660634 8168 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:19.660749 8168 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:19.660930 8168 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
W20260928 20:27:19.661407 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
W20260928 20:27:19.661492 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:19.661545 7755 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d; no voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c
I20260928 20:27:19.661634 8168 raft_consensus.cc:2726] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:27:20.439612 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:20.439332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:45215)
0928 20:27:20.439509 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:20.439562 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260928 20:27:20.513430 8170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:20.513212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:37079)
0928 20:27:20.513339 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:20.513383 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260928 20:27:20.639114 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:20.638886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:45579)
0928 20:27:20.639024 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:20.639066 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260928 20:27:21.440654 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:21.440255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:52211)
0928 20:27:21.440491 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:21.440579 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260928 20:27:21.514246 8173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:21.514001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:50099)
0928 20:27:21.514149 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:21.514198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260928 20:27:21.640146 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:21.639752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:46829)
0928 20:27:21.640053 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:21.640096 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":130,"threads_started":1}
I20260928 20:27:22.441613 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:22.441357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:38745)
0928 20:27:22.441512 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:22.441564 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260928 20:27:22.515362 8176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:22.515028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:50803)
0928 20:27:22.515238 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:22.515309 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":135,"threads_started":1}
I20260928 20:27:22.641078 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:22.640844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:59033)
0928 20:27:22.640984 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:22.641029 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260928 20:27:23.442420 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:23.442195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:38581)
0928 20:27:23.442327 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:23.442371 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260928 20:27:23.516500 8179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:23.516136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:43939)
0928 20:27:23.516376 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:23.516442 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260928 20:27:23.624464 8180 raft_consensus.cc:482] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:23.624564 8180 raft_consensus.cc:504] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f83e217fe36241e5a4df6ef68bca546c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 36547 } } peers { permanent_uuid: "caf84d20b28b46ed9d90060bbc716535" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 39131 } } peers { permanent_uuid: "b8f5239fd0144285b5eb27c32860b09d" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39215 } }
I20260928 20:27:23.624694 8180 leader_election.cc:288] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547), caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131)
W20260928 20:27:23.625142 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f83e217fe36241e5a4df6ef68bca546c (127.24.186.195:36547): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
W20260928 20:27:23.625245 7755 leader_election.cc:341] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf84d20b28b46ed9d90060bbc716535 (127.24.186.194:39131): Not found: Tablet not found: b14af2afba5f472dbbe34adbe184d4a3
I20260928 20:27:23.625281 7755 leader_election.cc:302] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8f5239fd0144285b5eb27c32860b09d; no voters: caf84d20b28b46ed9d90060bbc716535, f83e217fe36241e5a4df6ef68bca546c
I20260928 20:27:23.625419 8180 raft_consensus.cc:2726] T b14af2afba5f472dbbe34adbe184d4a3 P b8f5239fd0144285b5eb27c32860b09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:27:23.642012 8181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:23.641712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:37815)
0928 20:27:23.641923 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:23.641962 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":156,"threads_started":1}
I20260928 20:27:24.443272 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:24.443003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:37983)
0928 20:27:24.443159 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:24.443207 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 20:27:24.517371 8183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:24.517101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:33495)
0928 20:27:24.517275 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:24.517321 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260928 20:27:24.642985 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:24.642736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:59357)
0928 20:27:24.642898 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:24.642936 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260928 20:27:25.444181 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:25.443925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:57063)
0928 20:27:25.444084 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:25.444130 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260928 20:27:25.518594 8186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:25.518158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:47015)
0928 20:27:25.518456 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:25.518542 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":192,"threads_started":1}
I20260928 20:27:25.644068 8187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:25.643658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:50643)
0928 20:27:25.643951 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:25.644017 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":219,"threads_started":1}
I20260928 20:27:26.445214 8188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:26.444877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.193:36591)
0928 20:27:26.445116 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:26.445164 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260928 20:27:26.519548 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:26.519291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.194:42531)
0928 20:27:26.519441 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:26.519494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 20:27:26.639467 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7709
I20260928 20:27:26.644758 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 7872
I20260928 20:27:26.646090 8190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:26.644820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:38383 (local address 127.24.186.195:42657)
0928 20:27:26.645985 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:26.646028 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:38383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1142,"thread_start_us":67,"threads_started":1}
I20260928 20:27:26.649957 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8005
2026-09-28T20:27:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18213 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:27:26 dist-test-slave-k477 krb5kdc[8193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:27:26 dist-test-slave-k477 krb5kdc[8193](info): set up 2 sockets
Sep 28 20:27:26 dist-test-slave-k477 krb5kdc[8193](info): commencing operation
krb5kdc: starting...
W20260928 20:27:27.744675 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:27:27 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:27:27.871552 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:27.959633 8206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:27.959905 8206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:27.959931 8206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:27.961131 8206 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:27:27.961176 8206 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:27:27.961190 8206 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:27:27.961202 8206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:27.961216 8206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:27.961230 8206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:27.961242 8206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:27.961256 8206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:27.963075 8206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:27.963387 8206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:27.963662 8206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:27.970487 8212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:27.970487 8214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:27.970842 8206 server_base.cc:1034] running on GCE node
W20260928 20:27:27.971105 8211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:27.971614 8206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:27.972043 8206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:27.973244 8206 hybrid_clock.cc:648] HybridClock initialized: now 1790627247973215 us; error 49 us; skew 500 ppm
Sep 28 20:27:27 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627247, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:27.985636 8206 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:27.986274 8206 webserver.cc:460] Webserver started at http://127.24.186.254:45187/ using document root <none> and password file <none>
I20260928 20:27:27.986544 8206 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:27.986621 8206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:27.986789 8206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:27.988055 8206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f"
format_stamp: "Formatted at 2026-09-28 20:27:27 on dist-test-slave-k477"
server_key: "a7250a0a953f580725586a62b3149e55"
server_key_iv: "06e881b85521924ce8b66dd6ba7a0760"
server_key_version: "encryptionkey@0"
I20260928 20:27:27.988436 8206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f"
format_stamp: "Formatted at 2026-09-28 20:27:27 on dist-test-slave-k477"
server_key: "a7250a0a953f580725586a62b3149e55"
server_key_iv: "06e881b85521924ce8b66dd6ba7a0760"
server_key_version: "encryptionkey@0"
I20260928 20:27:27.989948 8206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:27.990919 8221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:27.991287 8206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:27.991432 8206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f"
format_stamp: "Formatted at 2026-09-28 20:27:27 on dist-test-slave-k477"
server_key: "a7250a0a953f580725586a62b3149e55"
server_key_iv: "06e881b85521924ce8b66dd6ba7a0760"
server_key_version: "encryptionkey@0"
I20260928 20:27:27.991621 8206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:28.001379 8206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:28.003723 8206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:28.003945 8206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:28.008556 8206 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:35161
I20260928 20:27:28.008605 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:35161 every 8 connection(s)
I20260928 20:27:28.008966 8206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:28.009847 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:28.012326 8274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap starting.
I20260928 20:27:28.012971 8274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:28.013288 8274 log.cc:824] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:28.014038 8274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: No bootstrap required, opened a new log
I20260928 20:27:28.017084 8274 raft_consensus.cc:348] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:28.017545 8274 raft_consensus.cc:374] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:28.017625 8274 raft_consensus.cc:729] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Initialized, Role: FOLLOWER
I20260928 20:27:28.017638 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8206
I20260928 20:27:28.017725 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:27:28.017946 25323 external_mini_cluster.cc:1428] Setting key 8d0f2020bf15722d0f724048993eb47f
I20260928 20:27:28.017881 8274 consensus_queue.cc:260] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:28.017992 8274 raft_consensus.cc:388] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:28.018029 8274 raft_consensus.cc:482] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:28.018088 8274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:28.019534 8274 raft_consensus.cc:504] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:28.019682 8274 leader_election.cc:302] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e838f3f876f64a7bb0755cdcbc8d2f0f; no voters:
I20260928 20:27:28.019901 8274 leader_election.cc:288] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:27:28.019986 8279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:28.020185 8274 sys_catalog.cc:564] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:28.020208 8279 raft_consensus.cc:686] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 LEADER]: Becoming Leader. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Running, Role: LEADER
I20260928 20:27:28.020419 8279 consensus_queue.cc:237] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:28.020876 8280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:27:28.020963 8280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:28.021102 8281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e838f3f876f64a7bb0755cdcbc8d2f0f. Latest consensus state: current_term: 1 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:27:28.021139 8281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:28.021576 8288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:28.022146 8288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:28.024017 8288 catalog_manager.cc:1347] Generated new cluster ID: 934e33c5bb5d4889b628362814d963fc
I20260928 20:27:28.024071 8288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:28.030582 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.018526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.24.186.254:35161)
0928 20:27:28.018736 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.018745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.018772 (+ 27us) server_negotiation.cc:407] Connection header received
0928 20:27:28.018895 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.018901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.018946 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.019063 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:28.019980 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.021482 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.022601 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.023095 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.023228 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.026181 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:28.026186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:28.026186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:28.026202 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:28.028338 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.028847 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.028852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.028853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.028992 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.029225 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.029225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.029226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.029571 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:28.030128 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.030206 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.030363 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
I20260928 20:27:28.053130 8288 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:27:28.053772 8288 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:28.065696 8288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Generated new TSK 0
I20260928 20:27:28.065953 8288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:27:28.103127 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:28.191522 8300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:28.191744 8300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:28.191771 8300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:28.193066 8300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:28.193118 8300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:28.193148 8300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:28.194761 8300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:28.195035 8300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:28.195279 8300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:28.200918 8306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:28.201071 8300 server_base.cc:1034] running on GCE node
W20260928 20:27:28.200925 8308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:28.200918 8305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:28.201386 8300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:28.201604 8300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:28.202996 8300 hybrid_clock.cc:648] HybridClock initialized: now 1790627248202952 us; error 40 us; skew 500 ppm
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:28.205992 8300 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:28.206524 8300 webserver.cc:460] Webserver started at http://127.24.186.193:33015/ using document root <none> and password file <none>
I20260928 20:27:28.206784 8300 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:28.206876 8300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:28.207028 8300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:28.208317 8300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "db5943ed75db4d0a9e7484342297aaf6"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "66293375c9886fcb5f81644388337026"
server_key_iv: "27da59a6a88b6cf29cfdf27e41157e7f"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.208709 8300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "db5943ed75db4d0a9e7484342297aaf6"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "66293375c9886fcb5f81644388337026"
server_key_iv: "27da59a6a88b6cf29cfdf27e41157e7f"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.210163 8300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:28.211246 8315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.211630 8300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:28.211781 8300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "db5943ed75db4d0a9e7484342297aaf6"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "66293375c9886fcb5f81644388337026"
server_key_iv: "27da59a6a88b6cf29cfdf27e41157e7f"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.211966 8300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:28.239596 8300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:28.241904 8300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:28.242066 8300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:28.242324 8300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:28.242645 8300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:28.242697 8300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.242743 8300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:28.242772 8300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.248853 8300 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35581
I20260928 20:27:28.248934 8428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35581 every 8 connection(s)
I20260928 20:27:28.249276 8300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:28.258752 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.250063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:33829 (local address 127.24.186.254:35161)
0928 20:27:28.250111 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.250113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.250336 (+ 223us) server_negotiation.cc:407] Connection header received
0928 20:27:28.250894 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.250895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.250907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.250961 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:28.251970 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.252487 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.253493 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.253701 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.253793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.256103 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:28.256105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:28.256105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:28.256112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:28.257291 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.257693 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.257695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.257695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.257753 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.258015 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.258016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.258017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.258131 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:28.258208 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.258482 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.258574 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:27:28.259008 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.250036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:33829)
0928 20:27:28.250213 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.250248 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.250766 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.251042 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.251046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.251638 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:28.251849 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.251871 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.252591 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.252592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.253392 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.253395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.253818 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.253819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.253948 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.254488 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:28.254497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:28.256008 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:28.257394 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.257401 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.257403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.257618 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.257822 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.257823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.257823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.257943 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.258199 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:28.258206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:28.258344 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.258460 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.258716 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260928 20:27:28.259466 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8300
I20260928 20:27:28.259500 8429 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:28.259613 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:27:28.259615 8429 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:28.259794 8429 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:28.259859 25323 external_mini_cluster.cc:1428] Setting key 4c03195fe3a245e175ab4e69a2195a0c
I20260928 20:27:28.260452 8237 ts_manager.cc:194] Registered new tserver with Master: db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581)
I20260928 20:27:28.261114 8237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:33829
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:27:28.286266 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:28.371464 8434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:28.371718 8434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:28.371745 8434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:28.373028 8434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:28.373085 8434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:28.373114 8434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:28.374763 8434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:28.375059 8434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:28.375349 8434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:28.381577 8440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:28.381747 8434 server_base.cc:1034] running on GCE node
W20260928 20:27:28.381577 8439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:28.381783 8442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:28.382313 8434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:28.382696 8434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:28.383944 8434 hybrid_clock.cc:648] HybridClock initialized: now 1790627248383881 us; error 62 us; skew 500 ppm
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:28.387193 8434 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:28.387861 8434 webserver.cc:460] Webserver started at http://127.24.186.194:36119/ using document root <none> and password file <none>
I20260928 20:27:28.388170 8434 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:28.388230 8434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:28.388370 8434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:28.389627 8434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "1cc3cc1861dd5c0c5f771e0ff4b64f77"
server_key_iv: "e8f01c000685222c9a27c95bd7184dce"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.390036 8434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "1cc3cc1861dd5c0c5f771e0ff4b64f77"
server_key_iv: "e8f01c000685222c9a27c95bd7184dce"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.391733 8434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:27:28.392729 8449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.393035 8434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:28.393165 8434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "1cc3cc1861dd5c0c5f771e0ff4b64f77"
server_key_iv: "e8f01c000685222c9a27c95bd7184dce"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.393272 8434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:28.412194 8434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:28.414290 8434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:28.414469 8434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:28.414835 8434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:28.415289 8434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:28.415359 8434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.415405 8434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:28.415431 8434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.421738 8434 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:35227
I20260928 20:27:28.421795 8562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:35227 every 8 connection(s)
I20260928 20:27:28.422148 8434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:27:28.422505 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8434
I20260928 20:27:28.422631 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:27:28.422859 25323 external_mini_cluster.cc:1428] Setting key 36e9e6324bf77626755d3425de9c655d
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
I20260928 20:27:28.433768 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.422956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:57597 (local address 127.24.186.254:35161)
0928 20:27:28.423017 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.423019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.423247 (+ 228us) server_negotiation.cc:407] Connection header received
0928 20:27:28.423961 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.423962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.423975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.424050 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:28.425021 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.425670 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.426962 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.427292 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.427373 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.430203 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:28.430205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:28.430206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:28.430215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:28.431998 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.432519 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.432520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.432521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.432590 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.432914 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.432915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.432915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.433057 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:28.433124 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.433625 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.433703 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:27:28.433916 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.422900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:57597)
0928 20:27:28.423106 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.423156 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.423848 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.424110 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.424114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.424716 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:28.424912 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.424917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.425771 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.425772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.426849 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.426851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.427382 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.427383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.427498 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.428191 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:28.428200 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:28.430086 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:28.432108 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.432116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.432118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.432417 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.432687 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.432688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.432689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.432808 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.433152 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:28.433158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:28.433379 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.433508 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.433656 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260928 20:27:28.434569 8563 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:28.434671 8563 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
I20260928 20:27:28.434890 8563 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:28.435350 8237 ts_manager.cc:194] Registered new tserver with Master: e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
I20260928 20:27:28.435750 8237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:57597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:27:28.451480 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:28.536763 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:28.536986 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:28.537011 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:28.538172 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:28.538216 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:28.538244 8568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:28.540025 8568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:28.540280 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:28.540516 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:28.545550 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:28.545557 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:28.545562 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:28.545843 8568 server_base.cc:1034] running on GCE node
I20260928 20:27:28.546059 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:28.546373 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:28.547557 8568 hybrid_clock.cc:648] HybridClock initialized: now 1790627248547544 us; error 27 us; skew 500 ppm
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:28.550225 8568 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:28.550738 8568 webserver.cc:460] Webserver started at http://127.24.186.195:33973/ using document root <none> and password file <none>
I20260928 20:27:28.550981 8568 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:28.551054 8568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:28.551209 8568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:27:28.552398 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "53afc05ed7eda5ac231abb52ac1febbf"
server_key_iv: "eef46c095abf9c8c7cc56015f603dcd8"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.552789 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "53afc05ed7eda5ac231abb52ac1febbf"
server_key_iv: "eef46c095abf9c8c7cc56015f603dcd8"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.554171 8568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:28.555032 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.555256 8568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:28.555382 8568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "53afc05ed7eda5ac231abb52ac1febbf"
server_key_iv: "eef46c095abf9c8c7cc56015f603dcd8"
server_key_version: "encryptionkey@0"
I20260928 20:27:28.555469 8568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:28.565171 8568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:28.567680 8568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:28.567880 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:28.568125 8568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:28.568454 8568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:28.568511 8568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.568555 8568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:28.568588 8568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:28.575115 8568 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38811
I20260928 20:27:28.575215 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38811 every 8 connection(s)
I20260928 20:27:28.575562 8568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:27:28.577080 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8568
I20260928 20:27:28.577235 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:27:28.577467 25323 external_mini_cluster.cc:1428] Setting key 7985ea74fdc78f86093091788635c195
Sep 28 20:27:28 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627248, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:28.586802 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.576810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:60887 (local address 127.24.186.254:35161)
0928 20:27:28.576877 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.576881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.577283 (+ 402us) server_negotiation.cc:407] Connection header received
0928 20:27:28.578203 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.578205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.578222 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.578306 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:28.579500 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.580139 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.581269 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.581460 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.581527 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.584186 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:28.584187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:28.584188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:28.584195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:28.585491 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.585915 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.585916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.585916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.585967 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.586184 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.586185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.586185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.586292 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:28.586341 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.586635 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.586707 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 20:27:28.586980 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.576680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:60887)
0928 20:27:28.576970 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.577062 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.577992 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.578438 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.578444 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.579162 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:28.579374 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.579380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.580255 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.580255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.581158 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.581161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.581609 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.581611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.581824 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.582473 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:28.582486 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:28.584072 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:28.585590 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.585597 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.585598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.585846 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.586023 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:28.586024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:28.586024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:28.586130 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:28.586353 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:28.586358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:28.586485 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.586576 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.586722 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":142,"threads_started":1}
I20260928 20:27:28.587544 8697 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:28.587631 8697 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:28.587788 8697 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:28.588450 8237 ts_manager.cc:194] Registered new tserver with Master: 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811)
I20260928 20:27:28.589212 8237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:60887
I20260928 20:27:28.589834 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:27:28.598639 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.591294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.24.186.254:35161)
0928 20:27:28.591393 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.591396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.591488 (+ 92us) server_negotiation.cc:407] Connection header received
0928 20:27:28.591571 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.591572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.591582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.591653 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:28.592291 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.592794 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.593678 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.593878 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.593955 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.595380 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:28.595386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:28.595387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:28.595394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:28.597169 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.597541 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.597542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.597542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.597597 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:28.597994 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:28.597994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:28.597995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:28.598156 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:28.598236 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.598410 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.598487 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:27:28.601753 8238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:27:28.603505 8238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:27:28.611757 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.607564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35581 (local address 127.0.0.1:60686)
0928 20:27:28.607925 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.607972 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.608049 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.608590 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.608592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.608594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:28.608659 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.608660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.609718 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.609719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.610720 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.610722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.611407 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.611408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.611495 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.611511 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.611604 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.611665 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":167,"threads_started":1}
I20260928 20:27:28.612171 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.607794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.24.186.193:35581)
0928 20:27:28.608050 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.608055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.608070 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:27:28.608391 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.608394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.608411 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.608486 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:28.608755 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.609604 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.610821 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.611307 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.611408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.611468 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.612032 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.612082 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":115,"threads_started":1}
I20260928 20:27:28.612800 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.607640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38811 (local address 127.0.0.1:35902)
0928 20:27:28.607925 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.607972 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.608049 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.608664 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.608665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.608668 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:28.608762 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.608764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.610248 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.610249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.611505 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.611508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.612552 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.612553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.612634 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.612638 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.612688 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.612730 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":53,"thread_start_us":96,"threads_started":1}
I20260928 20:27:28.612977 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.607983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:35227 (local address 127.0.0.1:57398)
0928 20:27:28.608134 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.608144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.608199 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.608947 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.608948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.608950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:28.608984 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.608986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.610294 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.610295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.611506 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.611508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.612697 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.612698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.612766 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.612770 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.612819 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.612895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260928 20:27:28.612910 8363 tablet_service.cc:1467] Processing CreateTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 (DEFAULT_TABLE table=test-table [id=3f35e9f868024493b089b90a765ed052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:28.613101 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.607735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.24.186.195:38811)
0928 20:27:28.608260 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.608264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.608273 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:27:28.608313 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.608315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.608330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.608405 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:28.608860 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.610093 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.611609 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.612344 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.612444 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.612500 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.612973 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.613023 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":98,"threads_started":1}
I20260928 20:27:28.613260 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.608237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57398 (local address 127.24.186.194:35227)
0928 20:27:28.608762 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.608765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.608773 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:27:28.608802 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.608804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.608839 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.608929 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:28.609048 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.610195 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.611597 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.612411 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.612607 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.612669 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.613104 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.613170 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":89,"threads_started":1}
I20260928 20:27:28.613296 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dd3f0291f4fd9ba23349736b3dd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:28.613729 8631 tablet_service.cc:1467] Processing CreateTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 (DEFAULT_TABLE table=test-table [id=3f35e9f868024493b089b90a765ed052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:28.613946 8497 tablet_service.cc:1467] Processing CreateTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 (DEFAULT_TABLE table=test-table [id=3f35e9f868024493b089b90a765ed052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:28.614046 8631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dd3f0291f4fd9ba23349736b3dd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:28.614223 8497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53dd3f0291f4fd9ba23349736b3dd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:28.616401 8714 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Bootstrap starting.
I20260928 20:27:28.617080 8714 tablet_bootstrap.cc:654] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:28.617506 8715 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Bootstrap starting.
I20260928 20:27:28.617692 8716 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Bootstrap starting.
I20260928 20:27:28.617717 8714 log.cc:824] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:28.618198 8715 tablet_bootstrap.cc:654] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:28.618407 8716 tablet_bootstrap.cc:654] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:28.618600 8714 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: No bootstrap required, opened a new log
I20260928 20:27:28.618682 8714 ts_tablet_manager.cc:1397] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:28.618822 8715 log.cc:824] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:28.619004 8716 log.cc:824] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:28.619834 8715 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: No bootstrap required, opened a new log
I20260928 20:27:28.619903 8715 ts_tablet_manager.cc:1397] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:28.620055 8716 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: No bootstrap required, opened a new log
I20260928 20:27:28.620116 8716 ts_tablet_manager.cc:1397] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:28.621133 8714 raft_consensus.cc:348] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.621318 8714 raft_consensus.cc:374] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:28.621353 8714 raft_consensus.cc:729] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5943ed75db4d0a9e7484342297aaf6, State: Initialized, Role: FOLLOWER
I20260928 20:27:28.621477 8714 consensus_queue.cc:260] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.621790 8714 ts_tablet_manager.cc:1428] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:27:28.621820 8429 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
I20260928 20:27:28.621934 8715 raft_consensus.cc:348] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.621922 8716 raft_consensus.cc:348] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.622071 8715 raft_consensus.cc:374] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:28.622128 8715 raft_consensus.cc:729] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b968176b554e6ea2dfc8e2c8ea6708, State: Initialized, Role: FOLLOWER
I20260928 20:27:28.622089 8716 raft_consensus.cc:374] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:28.622176 8716 raft_consensus.cc:729] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e714c162a3be47d1a3ebb0b06b0328b8, State: Initialized, Role: FOLLOWER
I20260928 20:27:28.622242 8715 consensus_queue.cc:260] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.622325 8716 consensus_queue.cc:260] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.622536 8697 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
I20260928 20:27:28.622596 8716 ts_tablet_manager.cc:1428] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 20:27:28.622700 8563 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
I20260928 20:27:28.622537 8715 ts_tablet_manager.cc:1428] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 20:27:28.672813 8564 tablet.cc:2369] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:27:28.750174 8430 tablet.cc:2369] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:27:28.776119 8720 raft_consensus.cc:482] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:28.776283 8720 raft_consensus.cc:504] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.776676 8720 leader_election.cc:288] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227), 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811)
I20260928 20:27:28.780370 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.777065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:50145 (local address 127.24.186.194:35227)
0928 20:27:28.777143 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.777146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.777157 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:27:28.777237 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.777239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.777264 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.777341 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:28.777520 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.778298 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.779553 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.780011 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.780093 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.780102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.780247 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.780285 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 20:27:28.780397 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.777032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:49557 (local address 127.24.186.195:38811)
0928 20:27:28.777094 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.777096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.777261 (+ 165us) server_negotiation.cc:407] Connection header received
0928 20:27:28.777346 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.777347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.777356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.777422 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:28.777607 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.778311 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.779572 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.780001 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.780093 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.780102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.780277 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.780317 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 20:27:28.780418 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.776925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38811 (local address 127.24.186.193:49557)
0928 20:27:28.777181 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.777191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.777279 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.777433 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.777434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.777435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:28.777518 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.777521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.778417 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.778418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.779475 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.779477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.780108 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.780109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.780190 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.780193 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.780268 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.780310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":81,"thread_start_us":64,"threads_started":1}
I20260928 20:27:28.780421 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.776925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:35227 (local address 127.24.186.193:50145)
0928 20:27:28.777087 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.777100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:28.777161 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:28.777348 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:28.777349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:28.777352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:28.777433 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.777434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.778421 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.778422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.779451 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:28.779454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.780093 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.780094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:28.780168 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.780172 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:28.780241 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:27:28.780283 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":86,"thread_start_us":78,"threads_started":1}
I20260928 20:27:28.780941 8517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dd3f0291f4fd9ba23349736b3dd41" candidate_uuid: "db5943ed75db4d0a9e7484342297aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" is_pre_election: true
I20260928 20:27:28.780995 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dd3f0291f4fd9ba23349736b3dd41" candidate_uuid: "db5943ed75db4d0a9e7484342297aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" is_pre_election: true
I20260928 20:27:28.781119 8651 raft_consensus.cc:2454] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5943ed75db4d0a9e7484342297aaf6 in term 0.
I20260928 20:27:28.781119 8517 raft_consensus.cc:2454] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db5943ed75db4d0a9e7484342297aaf6 in term 0.
I20260928 20:27:28.781364 8318 leader_election.cc:302] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6, e714c162a3be47d1a3ebb0b06b0328b8; no voters:
I20260928 20:27:28.781721 8720 raft_consensus.cc:2781] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:28.781893 8720 raft_consensus.cc:482] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:28.781934 8720 raft_consensus.cc:3037] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:28.782972 8720 raft_consensus.cc:504] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.783313 8720 leader_election.cc:288] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 1 election: Requested vote from peers e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227), 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811)
I20260928 20:27:28.783521 8651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dd3f0291f4fd9ba23349736b3dd41" candidate_uuid: "db5943ed75db4d0a9e7484342297aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
I20260928 20:27:28.783550 8517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53dd3f0291f4fd9ba23349736b3dd41" candidate_uuid: "db5943ed75db4d0a9e7484342297aaf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
I20260928 20:27:28.783617 8651 raft_consensus.cc:3037] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:28.783653 8517 raft_consensus.cc:3037] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:28.784474 8517 raft_consensus.cc:2454] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5943ed75db4d0a9e7484342297aaf6 in term 1.
I20260928 20:27:28.784483 8651 raft_consensus.cc:2454] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db5943ed75db4d0a9e7484342297aaf6 in term 1.
I20260928 20:27:28.784682 8318 leader_election.cc:302] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6, e714c162a3be47d1a3ebb0b06b0328b8; no voters:
I20260928 20:27:28.784795 8720 raft_consensus.cc:2781] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:28.784962 8720 raft_consensus.cc:686] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 LEADER]: Becoming Leader. State: Replica: db5943ed75db4d0a9e7484342297aaf6, State: Running, Role: LEADER
I20260928 20:27:28.785079 8720 consensus_queue.cc:237] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:28.785976 8237 catalog_manager.cc:5495] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "db5943ed75db4d0a9e7484342297aaf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20260928 20:27:28.794639 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.792180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60698 (local address 127.24.186.193:35581)
0928 20:27:28.792241 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:28.792244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:28.792257 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:27:28.792289 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:28.792290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:28.792307 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:28.792366 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:28.792487 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.792967 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.793747 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:28.793938 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:28.794082 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:28.794302 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:28.794410 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:28.794477 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:28.794513 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:27:28.794532 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:27:28.796597 8651 raft_consensus.cc:1261] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Refusing update from remote peer db5943ed75db4d0a9e7484342297aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:28.796594 8517 raft_consensus.cc:1261] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Refusing update from remote peer db5943ed75db4d0a9e7484342297aaf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:28.796974 8720 consensus_queue.cc:1035] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:28.797127 8725 consensus_queue.cc:1035] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:28.799322 8728 mvcc.cc:205] Tried to move back new op lower bound from 7334409211068133376 to 7334409211024371712. Current Snapshot: MvccSnapshot[applied={T|T < 7334409211068133376}]
I20260928 20:27:28.799875 8729 mvcc.cc:205] Tried to move back new op lower bound from 7334409211068133376 to 7334409211024371712. Current Snapshot: MvccSnapshot[applied={T|T < 7334409211068133376}]
I20260928 20:27:28.801481 8730 mvcc.cc:205] Tried to move back new op lower bound from 7334409211068133376 to 7334409211024371712. Current Snapshot: MvccSnapshot[applied={T|T < 7334409211068133376 or (T in {7334409211068133376})}]
I20260928 20:27:28.801896 8238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:28.802083 8238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:28.802982 8238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Sending DeleteTablet for 3 replicas of tablet e53dd3f0291f4fd9ba23349736b3dd41
I20260928 20:27:28.803330 8363 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:28 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:60686
I20260928 20:27:28.803381 8497 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:28 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:57398
I20260928 20:27:28.803602 8631 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:28 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35902
I20260928 20:27:28.803865 8739 tablet_replica.cc:330] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: stopping tablet replica
I20260928 20:27:28.803898 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8300
I20260928 20:27:28.804472 8739 raft_consensus.cc:2229] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:28.804618 8739 raft_consensus.cc:2258] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:28.805078 8739 ts_tablet_manager.cc:1905] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:28.805721 8738 tablet_replica.cc:330] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: stopping tablet replica
I20260928 20:27:28.805860 8738 raft_consensus.cc:2229] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:28.805990 8738 raft_consensus.cc:2258] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:28.806428 8738 ts_tablet_manager.cc:1905] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:28.807250 8739 ts_tablet_manager.cc:1918] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:28.807364 8739 log.cc:1196] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/e53dd3f0291f4fd9ba23349736b3dd41
I20260928 20:27:28.807653 8739 ts_tablet_manager.cc:1939] T e53dd3f0291f4fd9ba23349736b3dd41 P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting consensus metadata
I20260928 20:27:28.808151 8223 catalog_manager.cc:4841] TS 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): tablet e53dd3f0291f4fd9ba23349736b3dd41 (table test-table [id=3f35e9f868024493b089b90a765ed052]) successfully deleted
I20260928 20:27:28.808482 8738 ts_tablet_manager.cc:1918] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:28.808560 8738 log.cc:1196] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/e53dd3f0291f4fd9ba23349736b3dd41
I20260928 20:27:28.808820 8738 ts_tablet_manager.cc:1939] T e53dd3f0291f4fd9ba23349736b3dd41 P e714c162a3be47d1a3ebb0b06b0328b8: Deleting consensus metadata
I20260928 20:27:28.809528 8224 catalog_manager.cc:4841] TS e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): tablet e53dd3f0291f4fd9ba23349736b3dd41 (table test-table [id=3f35e9f868024493b089b90a765ed052]) successfully deleted
W20260928 20:27:28.810253 8224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:35581 (error 108)
I20260928 20:27:28.810411 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8434
I20260928 20:27:28.810791 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:28.810670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35581 (local address 127.0.0.1:60714)
0928 20:27:28.810725 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:28.810752 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:27:28.810997 8224 catalog_manager.cc:4584] TS db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e53dd3f0291f4fd9ba23349736b3dd41: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35581: connect: Connection refused (error 111)
I20260928 20:27:28.817308 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8568
I20260928 20:27:28.824754 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8206
I20260928 20:27:48.831106 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--webserver_interface=127.24.186.254
--webserver_port=45187
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:27:48.921356 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:48.921610 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:48.921638 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:48.923070 8741 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:27:48.923128 8741 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:27:48.923142 8741 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:27:48.923153 8741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:27:48.923167 8741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:48.923182 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:48.923239 8741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:27:48.923257 8741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:27:48.924988 8741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:48.925274 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:48.925529 8741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:48.935041 8747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:48.935072 8741 server_base.cc:1034] running on GCE node
W20260928 20:27:48.935046 8750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:48.935046 8748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:48.936033 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:48.937253 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:48.938571 8741 hybrid_clock.cc:648] HybridClock initialized: now 1790627268938612 us; error 156 us; skew 500 ppm
Sep 28 20:27:48 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627268, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:48.941449 8741 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:27:48.942025 8741 webserver.cc:460] Webserver started at http://127.24.186.254:45187/ using document root <none> and password file <none>
I20260928 20:27:48.942278 8741 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:48.942370 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:48.943897 8741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:48.944700 8757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:48.945025 8741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:48.945143 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f"
format_stamp: "Formatted at 2026-09-28 20:27:27 on dist-test-slave-k477"
server_key: "a7250a0a953f580725586a62b3149e55"
server_key_iv: "06e881b85521924ce8b66dd6ba7a0760"
server_key_version: "encryptionkey@0"
I20260928 20:27:48.945581 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:48.957854 8741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:48.960666 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:48.960866 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:48.965200 8741 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:35161
I20260928 20:27:48.965232 8809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:35161 every 8 connection(s)
I20260928 20:27:48.965641 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:27:48.967159 8810 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:27:48.967505 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8741
I20260928 20:27:48.967794 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:35581
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=33015
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:48.967962 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap starting.
I20260928 20:27:48.981825 8810 log.cc:824] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:48.984942 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:48.985357 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap complete.
I20260928 20:27:48.988724 8810 raft_consensus.cc:348] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:48.989079 8810 raft_consensus.cc:729] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Initialized, Role: FOLLOWER
I20260928 20:27:48.989250 8810 consensus_queue.cc:260] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:48.989374 8810 raft_consensus.cc:388] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:27:48.989441 8810 raft_consensus.cc:482] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:27:48.989483 8810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 1 FOLLOWER]: Advancing to term 2
I20260928 20:27:48.991047 8810 raft_consensus.cc:504] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:48.991189 8810 leader_election.cc:302] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e838f3f876f64a7bb0755cdcbc8d2f0f; no voters:
I20260928 20:27:48.991425 8810 leader_election.cc:288] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 20:27:48.991597 8814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 20:27:48.991744 8810 sys_catalog.cc:564] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:27:48.991875 8814 raft_consensus.cc:686] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 LEADER]: Becoming Leader. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Running, Role: LEADER
I20260928 20:27:48.992009 8814 consensus_queue.cc:237] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:27:48.992348 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:27:48.992442 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:48.992352 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e838f3f876f64a7bb0755cdcbc8d2f0f. Latest consensus state: current_term: 2 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:27:48.992587 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:27:48.992877 8823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:27:48.993824 8823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:27:48.993930 8823 catalog_manager.cc:1259] Loaded cluster ID: 934e33c5bb5d4889b628362814d963fc
I20260928 20:27:48.993963 8823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:27:48.995136 8823 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:27:48.995342 8823 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Loaded TSK: 0
I20260928 20:27:49.011943 8823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Generated new TSK 1
I20260928 20:27:49.012077 8823 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:27:49.065966 8812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:49.066220 8812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:49.066262 8812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:49.067488 8812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:49.067562 8812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:49.067612 8812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:27:49.069172 8812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:35581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=33015
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:49.069442 8812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:49.069676 8812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:49.074983 8838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:49.075012 8837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:49.075165 8840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:49.075606 8812 server_base.cc:1034] running on GCE node
I20260928 20:27:49.075903 8812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:49.076164 8812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:49.077519 8812 hybrid_clock.cc:648] HybridClock initialized: now 1790627269077474 us; error 42 us; skew 500 ppm
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:49.081040 8812 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:27:49.081593 8812 webserver.cc:460] Webserver started at http://127.24.186.193:33015/ using document root <none> and password file <none>
I20260928 20:27:49.081856 8812 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:49.081981 8812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:49.083590 8812 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:27:49.084550 8847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.084789 8812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:49.084894 8812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "db5943ed75db4d0a9e7484342297aaf6"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "66293375c9886fcb5f81644388337026"
server_key_iv: "27da59a6a88b6cf29cfdf27e41157e7f"
server_key_version: "encryptionkey@0"
I20260928 20:27:49.085387 8812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:49.097565 8812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:49.100281 8812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:49.100466 8812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:49.100745 8812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:49.101264 8854 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:27:49.102897 8812 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:27:49.102975 8812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:27:49.103016 8812 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:27:49.103952 8812 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:27:49.103998 8812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:49.104084 8854 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Bootstrap starting.
I20260928 20:27:49.111631 8812 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35581
I20260928 20:27:49.111842 8961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35581 every 8 connection(s)
I20260928 20:27:49.112200 8812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:27:49.115463 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8812
I20260928 20:27:49.115899 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:35227
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=36119
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:49.116652 8854 log.cc:824] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:49.119861 8854 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:27:49.120270 8854 tablet_bootstrap.cc:492] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Bootstrap complete.
I20260928 20:27:49.121575 8854 ts_tablet_manager.cc:1397] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260928 20:27:49.123524 8854 raft_consensus.cc:348] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:49.123975 8854 raft_consensus.cc:729] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5943ed75db4d0a9e7484342297aaf6, State: Initialized, Role: FOLLOWER
I20260928 20:27:49.124202 8854 consensus_queue.cc:260] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } }
I20260928 20:27:49.124567 8854 ts_tablet_manager.cc:1428] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:49.129088 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.112971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:52967)
0928 20:27:49.113177 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.113229 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.114036 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.117702 (+ 3666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.117707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.118062 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:49.118390 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.118398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.120141 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.120142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.121568 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.121572 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.122535 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.122537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.122680 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.123316 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:49.123325 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:49.125074 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:49.127157 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.127167 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.127169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.127414 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.127705 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.127706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.127707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.127884 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.128274 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:49.128280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:49.128508 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.128665 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.128866 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260928 20:27:49.129120 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.113060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:52967 (local address 127.24.186.254:35161)
0928 20:27:49.117011 (+ 3951us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.117023 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.117076 (+ 53us) server_negotiation.cc:407] Connection header received
0928 20:27:49.117232 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.117240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.117319 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.117509 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:49.118542 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.119994 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.121672 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.122314 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.122421 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.125210 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:49.125215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:49.125215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:49.125228 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:49.127044 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.127508 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.127512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.127512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.127625 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.127960 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.127961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.127961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.128196 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:49.128302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.128726 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.128866 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3883,"thread_start_us":113,"threads_started":1}
I20260928 20:27:49.130209 8962 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:49.130328 8962 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:49.130530 8962 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:49.131258 8774 ts_manager.cc:194] Registered new tserver with Master: db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581)
I20260928 20:27:49.132383 8774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:52967
I20260928 20:27:49.133253 8962 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
I20260928 20:27:49.139343 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.131989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.24.186.193:35581)
0928 20:27:49.132142 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.132147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.136123 (+ 3976us) server_negotiation.cc:407] Connection header received
0928 20:27:49.136233 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.136235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.136250 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.136355 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:49.136654 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.137335 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.138387 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.138886 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.138978 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.139041 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.139218 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.139263 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260928 20:27:49.139353 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.131951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35581 (local address 127.0.0.1:40738)
0928 20:27:49.135996 (+ 4045us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.136037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.136144 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.136364 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.136367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.136369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:49.136557 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.136561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.137431 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.137432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.138290 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.138293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.138995 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.138996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.139088 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.139101 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.139191 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.139253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3999,"thread_start_us":100,"threads_started":1}
I20260928 20:27:49.140082 8896 tablet_service.cc:1514] Processing DeleteTablet for tablet e53dd3f0291f4fd9ba23349736b3dd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:28 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:40738
I20260928 20:27:49.140641 8971 tablet_replica.cc:330] stopping tablet replica
I20260928 20:27:49.140770 8971 raft_consensus.cc:2229] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:49.140851 8971 raft_consensus.cc:2258] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:49.141168 8971 ts_tablet_manager.cc:1905] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:49.143643 8971 ts_tablet_manager.cc:1918] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:49.143769 8971 log.cc:1196] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal/wals/e53dd3f0291f4fd9ba23349736b3dd41
I20260928 20:27:49.144064 8971 ts_tablet_manager.cc:1939] T e53dd3f0291f4fd9ba23349736b3dd41 P db5943ed75db4d0a9e7484342297aaf6: Deleting consensus metadata
I20260928 20:27:49.144703 8760 catalog_manager.cc:4841] TS db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581): tablet e53dd3f0291f4fd9ba23349736b3dd41 (table test-table [id=3f35e9f868024493b089b90a765ed052]) successfully deleted
W20260928 20:27:49.224531 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:49.224771 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:49.224798 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:49.226085 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:49.226135 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:49.226163 8967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:27:49.228040 8967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=36119
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:49.228300 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:49.228526 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:49.234007 8977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:49.234015 8976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:49.234104 8967 server_base.cc:1034] running on GCE node
W20260928 20:27:49.234015 8979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:49.234424 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:49.234686 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:49.235877 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790627269235864 us; error 32 us; skew 500 ppm
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:49.238584 8967 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:27:49.239048 8967 webserver.cc:460] Webserver started at http://127.24.186.194:36119/ using document root <none> and password file <none>
I20260928 20:27:49.239303 8967 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:49.239380 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:49.240852 8967 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:49.241920 8986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.242244 8967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:49.242362 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "1cc3cc1861dd5c0c5f771e0ff4b64f77"
server_key_iv: "e8f01c000685222c9a27c95bd7184dce"
server_key_version: "encryptionkey@0"
I20260928 20:27:49.242790 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:49.254314 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:49.256891 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:49.257066 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:49.257318 8967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:49.257639 8967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:49.257671 8967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.257694 8967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:49.257742 8967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.263674 8967 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:35227
I20260928 20:27:49.263757 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:35227 every 8 connection(s)
I20260928 20:27:49.264147 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:49.274160 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 8967
I20260928 20:27:49.274498 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:38811
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=33973
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:27:49.274523 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.264974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:60055 (local address 127.24.186.254:35161)
0928 20:27:49.265051 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.265053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.265214 (+ 161us) server_negotiation.cc:407] Connection header received
0928 20:27:49.265738 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.265739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.265750 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.265818 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:49.266840 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.267337 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.268412 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.268629 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.268718 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.270975 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:49.270977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:49.270977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:49.270985 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:49.272972 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.273483 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.273485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.273486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.273555 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.273829 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.273830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.273831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.273958 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:49.274031 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.274317 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.274423 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:27:49.274772 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.264931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:60055)
0928 20:27:49.265090 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.265131 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.265623 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.265908 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.265911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.266494 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:49.266711 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.266716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.267450 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.267451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.268306 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.268309 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.268729 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.268730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.268873 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.269405 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:49.269412 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:49.270876 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:49.273107 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.273116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.273119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.273382 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.273633 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.273634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.273634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.273747 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.274032 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:49.274039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:49.274184 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.274299 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.274490 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260928 20:27:49.275312 9100 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:49.275434 9100 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:49.275630 9100 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:49.276111 8774 ts_manager.cc:194] Registered new tserver with Master: e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
I20260928 20:27:49.276504 8774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:60055
W20260928 20:27:49.365011 9103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:27:49.365293 9103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:27:49.365338 9103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:27:49.366636 9103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:27:49.366721 9103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:27:49.366771 9103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:27:49.368389 9103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:38811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=33973
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:27:49.368685 9103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:27:49.368942 9103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:27:49.374907 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:27:49.374837 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:49.374899 9103 server_base.cc:1034] running on GCE node
W20260928 20:27:49.374837 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:27:49.375334 9103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:27:49.375572 9103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:27:49.376731 9103 hybrid_clock.cc:648] HybridClock initialized: now 1790627269376667 us; error 85 us; skew 500 ppm
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:27:49.379401 9103 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:27:49.379935 9103 webserver.cc:460] Webserver started at http://127.24.186.195:33973/ using document root <none> and password file <none>
I20260928 20:27:49.380175 9103 fs_manager.cc:359] Metadata directory not provided
I20260928 20:27:49.380277 9103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:27:49.381804 9103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:49.382540 9118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.382746 9103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:27:49.382870 9103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "53afc05ed7eda5ac231abb52ac1febbf"
server_key_iv: "eef46c095abf9c8c7cc56015f603dcd8"
server_key_version: "encryptionkey@0"
I20260928 20:27:49.383248 9103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:27:49.402730 9103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:27:49.405212 9103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:27:49.405398 9103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:27:49.405650 9103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:27:49.405975 9103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:27:49.406028 9103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.406070 9103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:27:49.406095 9103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:27:49.412740 9103 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38811
I20260928 20:27:49.412792 9231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38811 every 8 connection(s)
I20260928 20:27:49.413166 9103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:27:49 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627269, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:27:49.420852 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9103
I20260928 20:27:49.424119 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.413988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:37285 (local address 127.24.186.254:35161)
0928 20:27:49.414060 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.414062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.414294 (+ 232us) server_negotiation.cc:407] Connection header received
0928 20:27:49.414838 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.414839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.414852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.414901 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:49.415923 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.416428 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.417686 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.417916 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.418006 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.420673 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:49.420675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:49.420675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:49.420682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:49.422339 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.422893 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.422894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.422895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.422963 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.423231 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.423232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.423232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.423353 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:49.423587 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.423928 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.424022 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:27:49.424350 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.413975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:37285)
0928 20:27:49.414159 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.414196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.414720 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.414976 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.414979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.415563 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:27:49.415768 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.415773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.416530 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.416531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.417579 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.417585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.418016 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.418018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.418156 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.418735 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:27:49.418744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:27:49.420563 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:27:49.422511 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.422520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.422523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.422793 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.423038 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:27:49.423039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:27:49.423039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:27:49.423159 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:27:49.423576 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:27:49.423584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:27:49.423743 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.423907 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.424083 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260928 20:27:49.424835 9232 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35161
I20260928 20:27:49.424955 9232 heartbeater.cc:464] Registering TS with master...
I20260928 20:27:49.425151 9232 heartbeater.cc:511] Master 127.24.186.254:35161 requested a full tablet report, sending...
I20260928 20:27:49.426049 8774 ts_manager.cc:194] Registered new tserver with Master: 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811)
I20260928 20:27:49.426546 8774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:37285
I20260928 20:27:49.428666 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.421394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.24.186.254:35161)
0928 20:27:49.421554 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.421556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.421676 (+ 120us) server_negotiation.cc:407] Connection header received
0928 20:27:49.421752 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.421753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.421767 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.421873 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:49.422474 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.422987 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.423772 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.424030 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.424119 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.425861 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:49.425863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:49.425863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:49.425871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:49.427430 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.427842 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.427843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.427844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.427906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.428122 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.428123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.428123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.428242 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:49.428428 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.428466 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.428554 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":53,"threads_started":1}
I20260928 20:27:49.429580 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:27:49.432265 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.429939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.24.186.254:35161)
0928 20:27:49.429995 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.429998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.430124 (+ 126us) server_negotiation.cc:407] Connection header received
0928 20:27:49.430163 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.430164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.430206 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.430294 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:49.430378 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.430860 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.431576 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.431787 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.431910 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.432039 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:49.432084 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 20:27:49.432212 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53322: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 20:27:49.432312 9235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53322: token expired
W20260928 20:27:49.432492 8703 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.24.186.254:35161)
I20260928 20:27:49.439111 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.432832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.24.186.254:35161)
0928 20:27:49.432897 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.432899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.432906 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:27:49.432925 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.432926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.432938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.432983 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:27:49.433456 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.433907 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.434611 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.434795 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.434853 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.436246 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:27:49.436248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:27:49.436248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:27:49.436255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:27:49.437730 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.438104 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.438105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.438105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.438166 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:27:49.438346 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:27:49.438346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:27:49.438347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:27:49.438589 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:27:49.438675 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.438837 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.438906 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:27:49.440707 8774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:27:49.442757 8774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:27:49.446004 8896 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa (DEFAULT_TABLE table=test-table [id=b6ad481c00fb44d78c37902b7f4ab39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:49.446251 8896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:49.448058 8854 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Bootstrap starting.
I20260928 20:27:49.448401 8854 tablet_bootstrap.cc:654] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:49.449008 8854 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: No bootstrap required, opened a new log
I20260928 20:27:49.449062 8854 ts_tablet_manager.cc:1397] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 20:27:49.449174 8854 raft_consensus.cc:348] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.449260 8854 raft_consensus.cc:374] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:49.449306 8854 raft_consensus.cc:729] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5943ed75db4d0a9e7484342297aaf6, State: Initialized, Role: FOLLOWER
I20260928 20:27:49.449350 8854 consensus_queue.cc:260] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.449435 8854 ts_tablet_manager.cc:1428] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 20:27:49.449779 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.445475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38811 (local address 127.0.0.1:38856)
0928 20:27:49.445541 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.445556 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.445601 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.446043 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.446044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.446046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:49.446101 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.446102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.447351 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.447353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.448377 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.448380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.449507 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.449508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.449581 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.449585 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.449610 (+ 25us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.449704 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 20:27:49.450057 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.445538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:35227 (local address 127.0.0.1:60466)
0928 20:27:49.445684 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.445695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.445752 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.446548 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.446549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.446551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:49.446603 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.446604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.447649 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.447650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.448870 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.448873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.449819 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.449820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.449878 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.449882 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.449940 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.449982 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260928 20:27:49.450068 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.445867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.24.186.194:35227)
0928 20:27:49.446073 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.446079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.446088 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:27:49.446429 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.446432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.446449 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.446550 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:49.446670 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.447544 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.448950 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.449742 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.449822 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.449879 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.449955 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.449991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260928 20:27:49.450347 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.445556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.24.186.195:38811)
0928 20:27:49.445823 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.445829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.445840 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:27:49.445886 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.445888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.445905 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.446047 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:49.446191 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.447262 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.448549 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.449408 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.450100 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.450173 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.450203 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.450253 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":125,"threads_started":1}
I20260928 20:27:49.450917 9034 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa (DEFAULT_TABLE table=test-table [id=b6ad481c00fb44d78c37902b7f4ab39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:49.450915 9166 tablet_service.cc:1467] Processing CreateTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa (DEFAULT_TABLE table=test-table [id=b6ad481c00fb44d78c37902b7f4ab39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:27:49.451256 9034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:49.451261 9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:27:49.455256 9243 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Bootstrap starting.
I20260928 20:27:49.456063 9243 tablet_bootstrap.cc:654] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:49.456617 9243 log.cc:824] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:49.456959 9242 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Bootstrap starting.
I20260928 20:27:49.457402 9243 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: No bootstrap required, opened a new log
I20260928 20:27:49.457463 9243 ts_tablet_manager.cc:1397] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:49.457515 9242 tablet_bootstrap.cc:654] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Neither blocks nor log segments found. Creating new log.
I20260928 20:27:49.457952 9242 log.cc:824] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Log is configured to *not* fsync() on all Append() calls
I20260928 20:27:49.458626 9242 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: No bootstrap required, opened a new log
I20260928 20:27:49.458702 9242 ts_tablet_manager.cc:1397] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:27:49.459021 9243 raft_consensus.cc:348] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.459163 9243 raft_consensus.cc:374] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:49.459185 9243 raft_consensus.cc:729] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b968176b554e6ea2dfc8e2c8ea6708, State: Initialized, Role: FOLLOWER
I20260928 20:27:49.459324 9243 consensus_queue.cc:260] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.459607 9243 ts_tablet_manager.cc:1428] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:27:49.459745 9232 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
I20260928 20:27:49.460641 9242 raft_consensus.cc:348] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.460773 9242 raft_consensus.cc:374] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:27:49.460794 9242 raft_consensus.cc:729] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e714c162a3be47d1a3ebb0b06b0328b8, State: Initialized, Role: FOLLOWER
I20260928 20:27:49.460937 9242 consensus_queue.cc:260] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.461220 9242 ts_tablet_manager.cc:1428] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 20:27:49.461220 9100 heartbeater.cc:503] Master 127.24.186.254:35161 was elected leader, sending a full tablet report...
W20260928 20:27:49.514792 9101 tablet.cc:2369] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:27:49.613308 8963 tablet.cc:2369] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:27:49.639916 9247 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:27:49.640045 9247 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.640492 9247 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581)
I20260928 20:27:49.643338 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.640765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:46797 (local address 127.24.186.195:38811)
0928 20:27:49.640821 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.640823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.640833 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:27:49.640889 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.640890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.640905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.640970 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:49.641127 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.641605 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.642580 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.642966 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.643049 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.643059 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.643225 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.643264 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:27:49.643347 9102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.640700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38811 (local address 127.24.186.194:46797)
0928 20:27:49.640759 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.640771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.640819 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.640976 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.640977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.640980 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:49.641054 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.641056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.641704 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.641706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.642469 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.642472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.643049 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.643050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.643136 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.643139 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.643216 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.643263 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 20:27:49.643522 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.640724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35581 (local address 127.24.186.194:35973)
0928 20:27:49.640882 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.640893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:27:49.640955 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:27:49.641115 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:27:49.641116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:27:49.641117 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:27:49.641187 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.641189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.641853 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.641854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.642611 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:27:49.642614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.643234 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.643235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:27:49.643316 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.643320 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:27:49.643398 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 20:27:49.643438 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260928 20:27:49.643515 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.640758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:35973 (local address 127.24.186.193:35581)
0928 20:27:49.640888 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.640890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.640940 (+ 50us) server_negotiation.cc:407] Connection header received
0928 20:27:49.641029 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.641030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.641039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.641113 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:27:49.641262 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.641755 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.642725 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.643114 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.643237 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.643246 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.643408 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.643449 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260928 20:27:49.643748 9186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddf89a01d9043d09fd7cec0cbcf36fa" candidate_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" is_pre_election: true
I20260928 20:27:49.643859 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddf89a01d9043d09fd7cec0cbcf36fa" candidate_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5943ed75db4d0a9e7484342297aaf6" is_pre_election: true
I20260928 20:27:49.644140 8916 raft_consensus.cc:2454] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e714c162a3be47d1a3ebb0b06b0328b8 in term 0.
I20260928 20:27:49.644150 9186 raft_consensus.cc:2454] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e714c162a3be47d1a3ebb0b06b0328b8 in term 0.
I20260928 20:27:49.644456 8989 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6, e714c162a3be47d1a3ebb0b06b0328b8; no voters:
I20260928 20:27:49.644608 9247 raft_consensus.cc:2781] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:27:49.644688 9247 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:27:49.644719 9247 raft_consensus.cc:3037] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:49.645455 9247 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.645605 9247 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [CANDIDATE]: Term 1 election: Requested vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581)
I20260928 20:27:49.645766 9186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddf89a01d9043d09fd7cec0cbcf36fa" candidate_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
I20260928 20:27:49.645781 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ddf89a01d9043d09fd7cec0cbcf36fa" candidate_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db5943ed75db4d0a9e7484342297aaf6"
I20260928 20:27:49.645885 9186 raft_consensus.cc:3037] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:49.645882 8916 raft_consensus.cc:3037] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:27:49.646525 8916 raft_consensus.cc:2454] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e714c162a3be47d1a3ebb0b06b0328b8 in term 1.
I20260928 20:27:49.646735 9186 raft_consensus.cc:2454] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e714c162a3be47d1a3ebb0b06b0328b8 in term 1.
I20260928 20:27:49.646763 8989 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6, e714c162a3be47d1a3ebb0b06b0328b8; no voters:
I20260928 20:27:49.646847 9247 raft_consensus.cc:2781] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:27:49.647008 9247 raft_consensus.cc:686] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 LEADER]: Becoming Leader. State: Replica: e714c162a3be47d1a3ebb0b06b0328b8, State: Running, Role: LEADER
I20260928 20:27:49.647131 9247 consensus_queue.cc:237] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:27:49.648111 8774 catalog_manager.cc:5495] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } health_report { overall_health: UNKNOWN } } }
W20260928 20:27:49.663782 9233 tablet.cc:2369] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:27:49.680672 9240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.678193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.24.186.194:35227)
0928 20:27:49.678273 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 20:27:49.678276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:27:49.678287 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:27:49.678326 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:27:49.678327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:27:49.678348 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:27:49.678433 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:27:49.678575 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.679061 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.679833 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:27:49.680032 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:27:49.680123 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:27:49.680299 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:27:49.680413 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:27:49.680509 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:27:49.680528 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 20:27:49.680549 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:27:49.682653 8916 raft_consensus.cc:1261] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Refusing update from remote peer e714c162a3be47d1a3ebb0b06b0328b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:49.682682 9186 raft_consensus.cc:1261] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Refusing update from remote peer e714c162a3be47d1a3ebb0b06b0328b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:27:49.683000 9247 consensus_queue.cc:1035] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:49.683178 9250 consensus_queue.cc:1035] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:27:49.685216 9253 mvcc.cc:205] Tried to move back new op lower bound from 7334409296617029632 to 7334409296475348992. Current Snapshot: MvccSnapshot[applied={T|T < 7334409296617029632}]
I20260928 20:27:49.685819 9254 mvcc.cc:205] Tried to move back new op lower bound from 7334409296617029632 to 7334409296475348992. Current Snapshot: MvccSnapshot[applied={T|T < 7334409296617029632}]
I20260928 20:27:49.685844 9255 mvcc.cc:205] Tried to move back new op lower bound from 7334409296617029632 to 7334409296475348992. Current Snapshot: MvccSnapshot[applied={T|T < 7334409296617029632}]
I20260928 20:27:49.687916 8774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:49.688010 8774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:27:49.688838 8774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Sending DeleteTablet for 3 replicas of tablet 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:27:49.689236 9166 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:49 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38856
I20260928 20:27:49.689234 9034 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:49 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:60466
I20260928 20:27:49.689591 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8812
I20260928 20:27:49.689621 8896 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:27:49 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:40738
I20260928 20:27:49.689644 9264 tablet_replica.cc:330] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: stopping tablet replica
I20260928 20:27:49.689759 9264 raft_consensus.cc:2229] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:27:49.689865 9264 raft_consensus.cc:2258] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:49.690341 9264 ts_tablet_manager.cc:1905] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:49.692653 9264 ts_tablet_manager.cc:1918] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:49.692744 9264 log.cc:1196] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:27:49.693080 9264 ts_tablet_manager.cc:1939] T 7ddf89a01d9043d09fd7cec0cbcf36fa P 73b968176b554e6ea2dfc8e2c8ea6708: Deleting consensus metadata
I20260928 20:27:49.693532 9263 tablet_replica.cc:330] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: stopping tablet replica
I20260928 20:27:49.693650 9263 raft_consensus.cc:2229] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:27:49.693596 8759 catalog_manager.cc:4841] TS 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): tablet 7ddf89a01d9043d09fd7cec0cbcf36fa (table test-table [id=b6ad481c00fb44d78c37902b7f4ab39d]) successfully deleted
I20260928 20:27:49.693799 9263 raft_consensus.cc:2258] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:27:49.694275 9263 ts_tablet_manager.cc:1905] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:27:49.696270 9263 ts_tablet_manager.cc:1918] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:27:49.696345 9263 log.cc:1196] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:27:49.696617 9263 ts_tablet_manager.cc:1939] T 7ddf89a01d9043d09fd7cec0cbcf36fa P e714c162a3be47d1a3ebb0b06b0328b8: Deleting consensus metadata
I20260928 20:27:49.697052 8760 catalog_manager.cc:4841] TS e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): tablet 7ddf89a01d9043d09fd7cec0cbcf36fa (table test-table [id=b6ad481c00fb44d78c37902b7f4ab39d]) successfully deleted
W20260928 20:27:49.697599 8760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:35581 (error 108)
I20260928 20:27:49.697928 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8967
I20260928 20:27:49.698120 9238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:27:49.697976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35581 (local address 127.0.0.1:40740)
0928 20:27:49.698037 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:27:49.698079 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 20:27:49.698760 8760 catalog_manager.cc:4584] TS db5943ed75db4d0a9e7484342297aaf6 (127.24.186.193:35581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ddf89a01d9043d09fd7cec0cbcf36fa: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35581: connect: Connection refused (error 111)
I20260928 20:27:49.703594 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9103
I20260928 20:27:49.708606 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 8741
I20260928 20:28:09.714430 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--webserver_interface=127.24.186.254
--webserver_port=45187
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:09.802788 9266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:09.803035 9266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:09.803059 9266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:09.804428 9266 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 20:28:09.804481 9266 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 20:28:09.804493 9266 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 20:28:09.804505 9266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:09.804519 9266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:09.804533 9266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:09.804579 9266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:09.804610 9266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:09.806228 9266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=45187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:09.806530 9266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:09.806811 9266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:09.812592 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:09.811901 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:09.811901 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:09.813527 9266 server_base.cc:1034] running on GCE node
I20260928 20:28:09.813874 9266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:09.814338 9266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:09.815572 9266 hybrid_clock.cc:648] HybridClock initialized: now 1790627289815548 us; error 59 us; skew 500 ppm
Sep 28 20:28:09 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627289, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:09.818333 9266 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:09.818809 9266 webserver.cc:460] Webserver started at http://127.24.186.254:45187/ using document root <none> and password file <none>
I20260928 20:28:09.819064 9266 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:09.819136 9266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:09.820709 9266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:09.821628 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:09.821983 9266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:09.822252 9266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f"
format_stamp: "Formatted at 2026-09-28 20:27:27 on dist-test-slave-k477"
server_key: "a7250a0a953f580725586a62b3149e55"
server_key_iv: "06e881b85521924ce8b66dd6ba7a0760"
server_key_version: "encryptionkey@0"
I20260928 20:28:09.822703 9266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:09.833491 9266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:09.836141 9266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:09.836431 9266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:09.840646 9266 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:35161
I20260928 20:28:09.840682 9333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:35161 every 8 connection(s)
I20260928 20:28:09.841073 9266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:09.842563 9334 sys_catalog.cc:263] Verifying existing consensus state
I20260928 20:28:09.843219 9334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap starting.
I20260928 20:28:09.850378 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9266
I20260928 20:28:09.850736 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:35581
--local_ip_for_outbound_sockets=127.24.186.193
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=33015
--webserver_interface=127.24.186.193
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:28:09.852027 9334 log.cc:824] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:09.854480 9334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:28:09.854818 9334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Bootstrap complete.
I20260928 20:28:09.857292 9334 raft_consensus.cc:348] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:28:09.857573 9334 raft_consensus.cc:729] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Initialized, Role: FOLLOWER
I20260928 20:28:09.857792 9334 consensus_queue.cc:260] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:28:09.857894 9334 raft_consensus.cc:388] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:09.857949 9334 raft_consensus.cc:482] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:09.857999 9334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 2 FOLLOWER]: Advancing to term 3
I20260928 20:28:09.859331 9334 raft_consensus.cc:504] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:28:09.859476 9334 leader_election.cc:302] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e838f3f876f64a7bb0755cdcbc8d2f0f; no voters:
I20260928 20:28:09.859716 9334 leader_election.cc:288] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 20:28:09.859754 9338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 3 FOLLOWER]: Leader election won for term 3
I20260928 20:28:09.859896 9338 raft_consensus.cc:686] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [term 3 LEADER]: Becoming Leader. State: Replica: e838f3f876f64a7bb0755cdcbc8d2f0f, State: Running, Role: LEADER
I20260928 20:28:09.860026 9334 sys_catalog.cc:564] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:09.859990 9338 consensus_queue.cc:237] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } }
I20260928 20:28:09.860272 9339 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:28:09.860381 9339 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:09.860575 9340 sys_catalog.cc:455] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e838f3f876f64a7bb0755cdcbc8d2f0f. Latest consensus state: current_term: 3 leader_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e838f3f876f64a7bb0755cdcbc8d2f0f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35161 } } }
I20260928 20:28:09.860662 9340 sys_catalog.cc:458] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:09.860787 9343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:09.861584 9343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:09.861716 9343 catalog_manager.cc:1259] Loaded cluster ID: 934e33c5bb5d4889b628362814d963fc
I20260928 20:28:09.861752 9343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:09.863143 9343 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:09.863324 9343 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Loaded TSK: 0
I20260928 20:28:09.863368 9343 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e838f3f876f64a7bb0755cdcbc8d2f0f: Loaded TSK: 1
F20260928 20:28:09.863600 9343 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f871c8c49ad google::LogMessage::Fail() at ??:0
@ 0x7f871c8c788a google::LogMessage::SendToLog() at ??:0
@ 0x7f871c8c44de google::LogMessage::Flush() at ??:0
@ 0x7f871c8c5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f871f031104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f871f02ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f87248221cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f872482248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8724822594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f87247e8c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f87247f2ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f871e7920d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f871e787954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f871f5eb6db start_thread at ??:0
@ 0x7f871b3c261f clone at ??:0
W20260928 20:28:09.955495 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:09.955796 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:09.955862 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:09.957074 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:09.957166 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:09.957216 9336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:09.958952 9336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:35581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=33015
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:09.959261 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:09.959667 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:09.964836 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:09.964821 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:09.964938 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:09.965078 9336 server_base.cc:1034] running on GCE node
I20260928 20:28:09.965337 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:09.965642 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:09.966830 9336 hybrid_clock.cc:648] HybridClock initialized: now 1790627289966812 us; error 35 us; skew 500 ppm
Sep 28 20:28:09 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627289, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:09.969689 9336 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:28:09.970178 9336 webserver.cc:460] Webserver started at http://127.24.186.193:33015/ using document root <none> and password file <none>
I20260928 20:28:09.970438 9336 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:09.970512 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:09.971953 9336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:09.972700 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:09.972883 9336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:09.972977 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "db5943ed75db4d0a9e7484342297aaf6"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "66293375c9886fcb5f81644388337026"
server_key_iv: "27da59a6a88b6cf29cfdf27e41157e7f"
server_key_version: "encryptionkey@0"
I20260928 20:28:09.973346 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:10.003225 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:10.005786 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:10.005970 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:10.006222 9336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:10.006714 9392 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 20:28:10.008805 9336 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 20:28:10.008966 9336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 20:28:10.009064 9336 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 20:28:10.009853 9336 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 20:28:10.009928 9336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 20:28:10.009989 9392 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Bootstrap starting.
I20260928 20:28:10.018342 9336 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35581
I20260928 20:28:10.018445 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35581 every 8 connection(s)
I20260928 20:28:10.018784 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:10.025735 9392 log.cc:824] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:10.026744 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9336
I20260928 20:28:10.027065 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:35227
--local_ip_for_outbound_sockets=127.24.186.194
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=36119
--webserver_interface=127.24.186.194
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:28:10.029031 9392 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 20:28:10.029445 9392 tablet_bootstrap.cc:492] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Bootstrap complete.
I20260928 20:28:10.030894 9392 ts_tablet_manager.cc:1397] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.009s
I20260928 20:28:10.032672 9392 raft_consensus.cc:348] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:10.033069 9392 raft_consensus.cc:729] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5943ed75db4d0a9e7484342297aaf6, State: Initialized, Role: FOLLOWER
I20260928 20:28:10.033265 9392 consensus_queue.cc:260] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:10.033607 9392 ts_tablet_manager.cc:1428] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 20:28:10.090384 9502 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:28:10.019533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:48787)
0928 20:28:10.019775 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.019884 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:10.021268 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:10.090131 (+ 68863us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":112,"threads_started":1}
W20260928 20:28:10.090747 9500 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:35161 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:35161: Client connection negotiation failed: client connection to 127.24.186.254:35161: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 20:28:10.091185 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.091028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:45263)
0928 20:28:10.091090 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.091143 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:28:10.092092 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.091954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:33387)
0928 20:28:10.092026 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.092065 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 20:28:10.092307 9500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 20:28:10.123966 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:10.124181 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:10.124209 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:10.125454 9504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:10.125502 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:10.125530 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:10.127066 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=36119
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:10.127336 9504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:10.127597 9504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:10.133067 9511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:10.133073 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:10.133235 9504 server_base.cc:1034] running on GCE node
W20260928 20:28:10.133080 9510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:10.133536 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:10.133831 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:10.135003 9504 hybrid_clock.cc:648] HybridClock initialized: now 1790627290134987 us; error 31 us; skew 500 ppm
Sep 28 20:28:10 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627290, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:10.138405 9504 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:28:10.138909 9504 webserver.cc:460] Webserver started at http://127.24.186.194:36119/ using document root <none> and password file <none>
I20260928 20:28:10.139163 9504 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:10.139232 9504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:10.140756 9504 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:10.141525 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.141798 9504 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:10.141914 9504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "e714c162a3be47d1a3ebb0b06b0328b8"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "1cc3cc1861dd5c0c5f771e0ff4b64f77"
server_key_iv: "e8f01c000685222c9a27c95bd7184dce"
server_key_version: "encryptionkey@0"
I20260928 20:28:10.142294 9504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:10.154210 9504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:10.156567 9504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:10.156744 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:10.156987 9504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:10.157307 9504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:10.157366 9504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.157428 9504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:10.157456 9504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.163784 9504 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:35227
I20260928 20:28:10.163878 9633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:35227 every 8 connection(s)
I20260928 20:28:10.164250 9504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:10.165539 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.165084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:35909)
0928 20:28:10.165309 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.165388 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
W20260928 20:28:10.165993 9634 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:35161 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:35161: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
I20260928 20:28:10.166298 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.166205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:59263)
0928 20:28:10.166254 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.166280 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:28:10.166604 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.166557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:46057)
0928 20:28:10.166577 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.166593 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 20:28:10.166736 9634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:28:10.174523 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9504
I20260928 20:28:10.174893 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:38811
--local_ip_for_outbound_sockets=127.24.186.195
--tserver_master_addrs=127.24.186.254:35161
--webserver_port=33973
--webserver_interface=127.24.186.195
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:10.264930 9637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:10.265161 9637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:10.265187 9637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:10.266481 9637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:10.266538 9637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:10.266567 9637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:10.268177 9637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:38811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=33973
--tserver_master_addrs=127.24.186.254:35161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:10.268458 9637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:10.268743 9637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:10.274464 9645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:10.274508 9642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:10.274507 9643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:10.274921 9637 server_base.cc:1034] running on GCE node
I20260928 20:28:10.275190 9637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:10.275491 9637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:10.276688 9637 hybrid_clock.cc:648] HybridClock initialized: now 1790627290276682 us; error 42 us; skew 500 ppm
Sep 28 20:28:10 dist-test-slave-k477 krb5kdc[8193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627290, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:10.279636 9637 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:28:10.280236 9637 webserver.cc:460] Webserver started at http://127.24.186.195:33973/ using document root <none> and password file <none>
I20260928 20:28:10.280488 9637 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:10.280562 9637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:10.282531 9637 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:28:10.283366 9652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.283601 9637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:10.283726 9637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "73b968176b554e6ea2dfc8e2c8ea6708"
format_stamp: "Formatted at 2026-09-28 20:27:28 on dist-test-slave-k477"
server_key: "53afc05ed7eda5ac231abb52ac1febbf"
server_key_iv: "eef46c095abf9c8c7cc56015f603dcd8"
server_key_version: "encryptionkey@0"
I20260928 20:28:10.284205 9637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:10.301151 9637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:10.303658 9637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:10.303918 9637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:10.304193 9637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:10.304529 9637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:10.304582 9637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.304625 9637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:10.304653 9637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:10.310848 9637 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:38811
I20260928 20:28:10.310927 9765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:38811 every 8 connection(s)
I20260928 20:28:10.311342 9637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:28:10.311910 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9637
I20260928 20:28:10.312836 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.312207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:55137)
0928 20:28:10.312558 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.312650 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
W20260928 20:28:10.313233 9766 heartbeater.cc:644] Failed to heartbeat to 127.24.186.254:35161 (0 consecutive failures): Network error: Failed to ping master at 127.24.186.254:35161: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
I20260928 20:28:10.313613 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.313427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:59115)
0928 20:28:10.313511 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.313575 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 20:28:10.314141 9768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:10.313957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:49743)
0928 20:28:10.314008 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:10.314108 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 20:28:10.314332 9766 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 20:28:11.093058 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.092776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:57225)
0928 20:28:11.092946 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:11.092996 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260928 20:28:11.167426 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.167150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:58897)
0928 20:28:11.167310 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:11.167365 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260928 20:28:11.315241 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.314879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:44891)
0928 20:28:11.315121 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:11.315179 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":101,"threads_started":1}
I20260928 20:28:11.834355 9773 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:11.834482 9773 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:11.834920 9773 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
Sep 28 20:28:11 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627289, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.194@KRBTEST.COM
Sep 28 20:28:11 dist-test-slave-k477 krb5kdc[8193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627289, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.195@KRBTEST.COM
I20260928 20:28:11.847429 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.835110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:38811 (local address 127.24.186.193:46501)
0928 20:28:11.835312 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:11.835325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:11.835385 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:11.836391 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:11.836393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:11.836808 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:11.837007 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:11.837013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.838470 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.838471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:11.839059 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:11.839061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.839697 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:11.839854 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:11.840566 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:11.840574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:11.843236 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:11.845544 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:11.845545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:11.845545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:11.845886 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:11.846187 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:11.846188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:11.846188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:11.846365 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:11.846852 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:11.846853 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:11.846985 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 20:28:11.847104 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 20:28:11.847328 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":17,"thread_start_us":102,"threads_started":1}
I20260928 20:28:11.847452 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.835243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:46501 (local address 127.24.186.195:38811)
0928 20:28:11.835403 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:11.835415 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:11.835438 (+ 23us) server_negotiation.cc:407] Connection header received
0928 20:28:11.836109 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:11.836112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:11.836172 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:11.836328 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:11.837182 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.838021 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839201 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.839603 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839727 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:11.843455 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:11.843459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:11.843460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:11.843470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:11.845461 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:11.845995 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:11.845998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:11.845998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:11.846102 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:11.846478 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:11.846479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:11.846479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:11.846748 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:11.846873 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:11.847169 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 20:28:11.847316 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260928 20:28:11.847466 9777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.835437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:35399 (local address 127.24.186.194:35227)
0928 20:28:11.835593 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:11.835600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:11.835619 (+ 19us) server_negotiation.cc:407] Connection header received
0928 20:28:11.836159 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:11.836161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:11.836201 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:11.836328 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:11.837168 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.838022 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839181 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.839606 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839764 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:11.843156 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:11.843168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:11.843168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:11.843182 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:11.845262 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:11.846183 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:11.846186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:11.846186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:11.846278 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:11.846492 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:11.846493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:11.846494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:11.846749 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:11.846853 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:11.847163 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 20:28:11.847321 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 20:28:11.847553 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:11.835191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:35227 (local address 127.24.186.193:35399)
0928 20:28:11.835409 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:11.835422 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:11.835488 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:11.836457 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:11.836458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:11.836807 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:11.837007 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:11.837013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.838236 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.838238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:11.839059 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:11.839061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:11.839740 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:11.839741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:11.839861 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:11.840564 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:11.840573 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:11.842842 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:11.845385 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:11.845399 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:11.845403 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:11.845886 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:11.846361 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:11.846362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:11.846363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:11.846431 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:11.846842 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:11.846849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:11.846985 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 20:28:11.847104 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 20:28:11.847419 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":160,"thread_start_us":57,"threads_started":1}
W20260928 20:28:11.849466 9387 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
W20260928 20:28:11.849602 9388 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:28:11.849661 9388 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6; no voters: 73b968176b554e6ea2dfc8e2c8ea6708, e714c162a3be47d1a3ebb0b06b0328b8
I20260928 20:28:11.849772 9773 raft_consensus.cc:2726] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:28:12.093783 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:12.093607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:32843)
0928 20:28:12.093698 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:12.093743 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 20:28:12.168689 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:12.168196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:47661)
0928 20:28:12.168568 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:12.168633 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":279,"threads_started":1}
I20260928 20:28:12.316344 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:12.316043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:36963)
0928 20:28:12.316221 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:12.316281 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260928 20:28:13.094913 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:13.094688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:60723)
0928 20:28:13.094829 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:13.094867 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260928 20:28:13.169658 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:13.169427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:55603)
0928 20:28:13.169569 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:13.169605 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260928 20:28:13.317198 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:13.316954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:53169)
0928 20:28:13.317096 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:13.317140 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260928 20:28:13.566076 9783 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:13.566196 9783 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:13.566411 9783 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
W20260928 20:28:13.566885 9387 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
W20260928 20:28:13.566967 9388 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:28:13.567025 9388 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6; no voters: 73b968176b554e6ea2dfc8e2c8ea6708, e714c162a3be47d1a3ebb0b06b0328b8
I20260928 20:28:13.567113 9783 raft_consensus.cc:2726] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:28:14.095901 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:14.095540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:49707)
0928 20:28:14.095730 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:14.095776 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260928 20:28:14.170490 9785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:14.170171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:54535)
0928 20:28:14.170383 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:14.170440 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260928 20:28:14.318562 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:14.318134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:52715)
0928 20:28:14.318356 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:14.318495 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
I20260928 20:28:15.096690 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:15.096438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:42623)
0928 20:28:15.096580 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:15.096633 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260928 20:28:15.171288 9788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:15.171009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:51901)
0928 20:28:15.171181 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:15.171239 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260928 20:28:15.319372 9789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:15.319143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:34197)
0928 20:28:15.319271 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:15.319312 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260928 20:28:15.732333 9790 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:15.732436 9790 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:15.732612 9790 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
W20260928 20:28:15.733043 9387 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
W20260928 20:28:15.733124 9388 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:28:15.733208 9388 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6; no voters: 73b968176b554e6ea2dfc8e2c8ea6708, e714c162a3be47d1a3ebb0b06b0328b8
I20260928 20:28:15.733285 9790 raft_consensus.cc:2726] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:28:16.097607 9791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:16.097262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:54395)
0928 20:28:16.097512 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:16.097559 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":161,"threads_started":1}
I20260928 20:28:16.172174 9792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:16.171940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:60197)
0928 20:28:16.172081 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:16.172127 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 20:28:16.320201 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:16.319962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:53395)
0928 20:28:16.320108 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:16.320154 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260928 20:28:17.098644 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:17.098386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:53965)
0928 20:28:17.098548 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:17.098595 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260928 20:28:17.173051 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:17.172822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:43815)
0928 20:28:17.172959 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:17.173003 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260928 20:28:17.321026 9796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:17.320783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:54159)
0928 20:28:17.320930 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:17.320974 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260928 20:28:18.099540 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:18.099281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:49247)
0928 20:28:18.099443 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:18.099492 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260928 20:28:18.173830 9798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:18.173570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:47653)
0928 20:28:18.173737 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:18.173782 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 20:28:18.321800 9799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:18.321576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:56877)
0928 20:28:18.321711 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:18.321753 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260928 20:28:19.100432 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:19.100151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:38809)
0928 20:28:19.100332 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:19.100384 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260928 20:28:19.174860 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:19.174569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:49595)
0928 20:28:19.174767 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:19.174811 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":87,"threads_started":1}
I20260928 20:28:19.322727 9802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:19.322453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:44927)
0928 20:28:19.322630 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:19.322675 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
I20260928 20:28:19.366695 9803 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:19.366799 9803 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:19.367041 9803 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
W20260928 20:28:19.367542 9388 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
W20260928 20:28:19.367617 9387 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:28:19.367660 9387 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6; no voters: 73b968176b554e6ea2dfc8e2c8ea6708, e714c162a3be47d1a3ebb0b06b0328b8
I20260928 20:28:19.367774 9803 raft_consensus.cc:2726] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:28:20.101286 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:20.101040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:36087)
0928 20:28:20.101191 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:20.101232 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 20:28:20.175709 9805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:20.175426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:53809)
0928 20:28:20.175578 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:20.175659 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":66,"threads_started":1}
I20260928 20:28:20.323668 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:20.323274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:45471)
0928 20:28:20.323567 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:20.323602 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":146,"threads_started":1}
I20260928 20:28:21.102206 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:21.101932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:42071)
0928 20:28:21.102115 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:21.102159 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260928 20:28:21.176692 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:21.176401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:41669)
0928 20:28:21.176590 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:21.176643 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260928 20:28:21.324610 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:21.324322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:51597)
0928 20:28:21.324512 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:21.324555 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260928 20:28:22.103087 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:22.102833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:55667)
0928 20:28:22.102984 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:22.103029 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260928 20:28:22.177474 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:22.177263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:35929)
0928 20:28:22.177385 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:22.177424 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260928 20:28:22.325428 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:22.325188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:41841)
0928 20:28:22.325339 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:22.325381 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260928 20:28:22.539091 9813 raft_consensus.cc:482] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:22.539247 9813 raft_consensus.cc:504] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73b968176b554e6ea2dfc8e2c8ea6708" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 38811 } } peers { permanent_uuid: "e714c162a3be47d1a3ebb0b06b0328b8" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 35227 } } peers { permanent_uuid: "db5943ed75db4d0a9e7484342297aaf6" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35581 } }
I20260928 20:28:22.539487 9813 leader_election.cc:288] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811), e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227)
W20260928 20:28:22.539927 9387 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 73b968176b554e6ea2dfc8e2c8ea6708 (127.24.186.195:38811): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
W20260928 20:28:22.539995 9388 leader_election.cc:341] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e714c162a3be47d1a3ebb0b06b0328b8 (127.24.186.194:35227): Not found: Tablet not found: 7ddf89a01d9043d09fd7cec0cbcf36fa
I20260928 20:28:22.540079 9388 leader_election.cc:302] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db5943ed75db4d0a9e7484342297aaf6; no voters: 73b968176b554e6ea2dfc8e2c8ea6708, e714c162a3be47d1a3ebb0b06b0328b8
I20260928 20:28:22.540167 9813 raft_consensus.cc:2726] T 7ddf89a01d9043d09fd7cec0cbcf36fa P db5943ed75db4d0a9e7484342297aaf6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 20:28:23.103951 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:23.103620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:34751)
0928 20:28:23.103791 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:23.103903 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260928 20:28:23.178268 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:23.178024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:50789)
0928 20:28:23.178168 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:23.178216 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260928 20:28:23.326135 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:23.325913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:44043)
0928 20:28:23.326049 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:23.326089 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260928 20:28:24.104699 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:24.104470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:58829)
0928 20:28:24.104602 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:24.104650 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260928 20:28:24.179194 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:24.178951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:38055)
0928 20:28:24.179089 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:24.179141 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260928 20:28:24.327054 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:24.326807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.195:59309)
0928 20:28:24.326960 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:24.327006 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260928 20:28:25.105662 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:25.105392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.193:38971)
0928 20:28:25.105578 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:25.105619 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
I20260928 20:28:25.180181 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:25.179909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35161 (local address 127.24.186.194:60861)
0928 20:28:25.180064 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:25.180118 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.254:35161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260928 20:28:25.312502 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9336
I20260928 20:28:25.319154 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9504
I20260928 20:28:25.324165 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9637
2026-09-28T20:28:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58674 ms)
[----------] 30 tests from SecurityITest (244401 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:25 dist-test-slave-k477 krb5kdc[9824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:25 dist-test-slave-k477 krb5kdc[9824](info): set up 2 sockets
Sep 28 20:28:25 dist-test-slave-k477 krb5kdc[9824](info): commencing operation
krb5kdc: starting...
W20260928 20:28:26.392364 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:26 dist-test-slave-k477 krb5kdc[9824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:26.525202 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40455
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:40455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:26.614159 9837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:26.614410 9837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:26.614449 9837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:26.615622 9837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:26.615691 9837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:26.615716 9837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:26.615746 9837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:26.615770 9837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:26.617293 9837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:40455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:40455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:26.617571 9837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:26.617812 9837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:26.623421 9842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:26.623440 9845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:26.623533 9843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:26.623564 9837 server_base.cc:1034] running on GCE node
I20260928 20:28:26.624003 9837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:26.624337 9837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:26.625535 9837 hybrid_clock.cc:648] HybridClock initialized: now 1790627306625520 us; error 55 us; skew 500 ppm
Sep 28 20:28:26 dist-test-slave-k477 krb5kdc[9824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627306, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:26.628764 9837 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:26.629262 9837 webserver.cc:460] Webserver started at http://127.24.186.254:40367/ using document root <none> and password file <none>
I20260928 20:28:26.629513 9837 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:26.629582 9837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:26.629732 9837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:26.630842 9837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "d13fec58601d4dd38d74fcafadbe59fa"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "fb167af40a085c21339cb5c198bb3af7"
server_key_iv: "116b433d1cd486a116a17c9775814905"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.631260 9837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "d13fec58601d4dd38d74fcafadbe59fa"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "fb167af40a085c21339cb5c198bb3af7"
server_key_iv: "116b433d1cd486a116a17c9775814905"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.632900 9837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:26.633715 9852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:26.633944 9837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:26.634058 9837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "d13fec58601d4dd38d74fcafadbe59fa"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "fb167af40a085c21339cb5c198bb3af7"
server_key_iv: "116b433d1cd486a116a17c9775814905"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.634146 9837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:26.658605 9837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:26.660925 9837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:26.661108 9837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:26.666937 9837 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:40455
I20260928 20:28:26.666967 9904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:40455 every 8 connection(s)
I20260928 20:28:26.667366 9837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:26.668354 9905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:26.670661 9905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: Bootstrap starting.
I20260928 20:28:26.671152 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9837
I20260928 20:28:26.671280 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:26.671449 9905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:26.671509 25323 external_mini_cluster.cc:1428] Setting key d13c50de2022760b19b69febb29110dd
I20260928 20:28:26.671854 9905 log.cc:824] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:26.673168 9905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: No bootstrap required, opened a new log
I20260928 20:28:26.676170 9905 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } }
I20260928 20:28:26.676379 9905 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:26.676416 9905 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13fec58601d4dd38d74fcafadbe59fa, State: Initialized, Role: FOLLOWER
I20260928 20:28:26.676549 9905 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } }
I20260928 20:28:26.676630 9905 raft_consensus.cc:388] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:26.676661 9905 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:26.676709 9905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:26.677464 9905 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } }
I20260928 20:28:26.677598 9905 leader_election.cc:302] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d13fec58601d4dd38d74fcafadbe59fa; no voters:
I20260928 20:28:26.677805 9905 leader_election.cc:288] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:26.677829 9910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:26.677954 9910 raft_consensus.cc:686] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [term 1 LEADER]: Becoming Leader. State: Replica: d13fec58601d4dd38d74fcafadbe59fa, State: Running, Role: LEADER
I20260928 20:28:26.678076 9905 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:26.678098 9910 consensus_queue.cc:237] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } }
I20260928 20:28:26.678483 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d13fec58601d4dd38d74fcafadbe59fa. Latest consensus state: current_term: 1 leader_uuid: "d13fec58601d4dd38d74fcafadbe59fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } } }
I20260928 20:28:26.678491 9912 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d13fec58601d4dd38d74fcafadbe59fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13fec58601d4dd38d74fcafadbe59fa" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 40455 } } }
I20260928 20:28:26.678690 9912 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:26.678678 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:26.678961 9919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:26.679553 9919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 20:28:26 dist-test-slave-k477 krb5kdc[9824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:26.682777 9919 catalog_manager.cc:1347] Generated new cluster ID: 6946f8ff8f4d4905834957ee5f1d9bc6
I20260928 20:28:26.682839 9919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:26.685436 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:26.672478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.24.186.254:40455)
0928 20:28:26.672649 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:26.672659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:26.672794 (+ 135us) server_negotiation.cc:407] Connection header received
0928 20:28:26.673751 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:26.673760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:26.673834 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:26.674066 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:26.675014 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.676048 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.677138 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.677564 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.677739 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:26.681500 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:26.681505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:26.681506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:26.681520 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:26.683679 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:26.684179 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:26.684182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:26.684183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:26.684287 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:26.684505 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:26.684505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:26.684506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:26.684806 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:26.684961 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:26.685092 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:28:26.685232 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 20:28:26.688931 9919 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:26.689445 9919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:26.698969 9919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: Generated new TSK 0
I20260928 20:28:26.699213 9919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:28:26.736024 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:26.823599 9931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:26.823856 9931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:26.823886 9931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:26.825057 9931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:26.825098 9931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:26.825125 9931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:26.826903 9931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:26.827116 9931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:26.827312 9931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:26.832551 9939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:26.832543 9937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:26.832540 9936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:26.832890 9931 server_base.cc:1034] running on GCE node
I20260928 20:28:26.833096 9931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:26.833369 9931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:26.834549 9931 hybrid_clock.cc:648] HybridClock initialized: now 1790627306834539 us; error 35 us; skew 500 ppm
Sep 28 20:28:26 dist-test-slave-k477 krb5kdc[9824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627306, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:26.837538 9931 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:28:26.838002 9931 webserver.cc:460] Webserver started at http://127.24.186.193:41483/ using document root <none> and password file <none>
I20260928 20:28:26.838292 9931 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:26.838347 9931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:26.838442 9931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:26.839519 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "f6a7b9b0103f49f5b482eee3a31f3323"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "95e99db61d7e9a356bcbef6283a5411b"
server_key_iv: "f82b0af20a8a52f8f6760bd06a76cb77"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.839902 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "f6a7b9b0103f49f5b482eee3a31f3323"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "95e99db61d7e9a356bcbef6283a5411b"
server_key_iv: "f82b0af20a8a52f8f6760bd06a76cb77"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.841312 9931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:26.842094 9946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:26.842278 9931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 20:28:26.842389 9931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "f6a7b9b0103f49f5b482eee3a31f3323"
format_stamp: "Formatted at 2026-09-28 20:28:26 on dist-test-slave-k477"
server_key: "95e99db61d7e9a356bcbef6283a5411b"
server_key_iv: "f82b0af20a8a52f8f6760bd06a76cb77"
server_key_version: "encryptionkey@0"
I20260928 20:28:26.842464 9931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:26.853334 9931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:26.855659 9931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:26.855880 9931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:26.856143 9931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:26.856467 9931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:26.856519 9931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:26.856560 9931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:26.856585 9931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:26.862855 9931 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:35391
I20260928 20:28:26.862902 10059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:35391 every 8 connection(s)
I20260928 20:28:26.863233 9931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
Sep 28 20:28:26 dist-test-slave-k477 krb5kdc[9824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627306, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:26.871881 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 9931
I20260928 20:28:26.872023 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:26.872241 25323 external_mini_cluster.cc:1428] Setting key bfc3b79c3754b01f41e1c548a98f6b31
I20260928 20:28:26.874759 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:26.864076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:47631 (local address 127.24.186.254:40455)
0928 20:28:26.864199 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:26.864202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:26.864492 (+ 290us) server_negotiation.cc:407] Connection header received
0928 20:28:26.865243 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:26.865245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:26.865262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:26.865317 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:26.866762 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.867302 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.868428 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.868645 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.868727 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:26.871383 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:26.871385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:26.871385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:26.871401 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:26.873152 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:26.873615 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:26.873616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:26.873616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:26.873681 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:26.873953 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:26.873954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:26.873955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:26.874081 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:26.874175 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:26.874553 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 20:28:26.874665 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 20:28:26.874987 10062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:26.863997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40455 (local address 127.24.186.193:47631)
0928 20:28:26.864251 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:26.864354 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:26.865100 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:26.865424 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:26.865429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:26.866180 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:26.866553 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:26.866565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.867412 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.867413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:26.868334 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:26.868337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:26.868733 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:26.868734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:26.868860 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:26.869402 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:26.869414 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:26.871270 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:26.873263 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:26.873273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:26.873275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:26.873519 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:26.873764 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:26.873765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:26.873765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:26.873881 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:26.874181 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:26.874190 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:26.874402 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 20:28:26.874531 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 20:28:26.874703 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
I20260928 20:28:26.875698 10060 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40455
I20260928 20:28:26.875864 10060 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:26.876076 10060 heartbeater.cc:511] Master 127.24.186.254:40455 requested a full tablet report, sending...
I20260928 20:28:26.876767 9869 ts_manager.cc:194] Registered new tserver with Master: f6a7b9b0103f49f5b482eee3a31f3323 (127.24.186.193:35391)
I20260928 20:28:26.877728 9869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:47631
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:28:26.903537 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:26.995736 10065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:26.995985 10065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:26.996012 10065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:26.997392 10065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:26.997447 10065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:26.997476 10065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:26.999045 10065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:26.999311 10065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:26.999588 10065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:27.004961 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.005038 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.005174 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:27.005080 10065 server_base.cc:1034] running on GCE node
I20260928 20:28:27.005409 10065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:27.006026 10065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:27.007303 10065 hybrid_clock.cc:648] HybridClock initialized: now 1790627307007228 us; error 112 us; skew 500 ppm
Sep 28 20:28:27 dist-test-slave-k477 krb5kdc[9824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:27.010125 10065 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:28:27.010675 10065 webserver.cc:460] Webserver started at http://127.24.186.194:37615/ using document root <none> and password file <none>
I20260928 20:28:27.010921 10065 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:27.010989 10065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:27.011135 10065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:27.012301 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "077326239da94903b6a02026bd44feb3"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "b91aac4a8f945401128230767ba316d5"
server_key_iv: "bb1b84265f0c8e9f2e75cbf16c2b690f"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.012669 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "077326239da94903b6a02026bd44feb3"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "b91aac4a8f945401128230767ba316d5"
server_key_iv: "bb1b84265f0c8e9f2e75cbf16c2b690f"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.014106 10065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:27.014966 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.015143 10065 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:27.015240 10065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "077326239da94903b6a02026bd44feb3"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "b91aac4a8f945401128230767ba316d5"
server_key_iv: "bb1b84265f0c8e9f2e75cbf16c2b690f"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.015601 10065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:27.034467 10065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:27.037302 10065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:27.037477 10065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:27.037770 10065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:27.038091 10065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:27.038146 10065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.038188 10065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:27.038216 10065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.044332 10065 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:34747
I20260928 20:28:27.044536 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:34747 every 8 connection(s)
I20260928 20:28:27.044773 10065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:27.049612 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10065
I20260928 20:28:27.049763 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:27.049962 25323 external_mini_cluster.cc:1428] Setting key 93308660a5be7e2b38a81a5c51893cff
Sep 28 20:28:27 dist-test-slave-k477 krb5kdc[9824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:27.055282 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.045634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:50257 (local address 127.24.186.254:40455)
0928 20:28:27.045700 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.045703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.045887 (+ 184us) server_negotiation.cc:407] Connection header received
0928 20:28:27.046405 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.046406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.046417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.046474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:27.047508 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.048102 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.049104 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.049295 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.049370 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.051873 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:27.051875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:27.051876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:27.051884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:27.053639 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.054202 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.054203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.054203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.054257 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.054585 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.054586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.054586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.054721 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:27.054778 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.055139 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.055213 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:28:27.055636 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.045613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40455 (local address 127.24.186.194:50257)
0928 20:28:27.045763 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.045800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.046282 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.046527 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.046530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.047210 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:27.047408 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.047412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.048182 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.048183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.049012 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.049014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.049409 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.049411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.049539 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.050162 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:27.050171 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:27.051737 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:27.053739 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:27.053747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:27.053750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:27.054103 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:27.054334 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:27.054335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:27.054336 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:27.054498 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:27.054796 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:27.054803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:27.054985 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.055116 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.055314 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260928 20:28:27.056175 10194 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40455
I20260928 20:28:27.056291 10194 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:27.056510 10194 heartbeater.cc:511] Master 127.24.186.254:40455 requested a full tablet report, sending...
I20260928 20:28:27.057323 9869 ts_manager.cc:194] Registered new tserver with Master: 077326239da94903b6a02026bd44feb3 (127.24.186.194:34747)
I20260928 20:28:27.057729 9869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:50257
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:28:27.079355 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:27.164640 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:27.164901 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:27.164939 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:27.166229 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:27.166332 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:27.166383 10199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:27.167985 10199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:40455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:27.168255 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:27.168491 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:27.173846 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.173859 10205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.173871 10207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:27.174093 10199 server_base.cc:1034] running on GCE node
I20260928 20:28:27.174324 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:27.174582 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:27.175793 10199 hybrid_clock.cc:648] HybridClock initialized: now 1790627307175783 us; error 48 us; skew 500 ppm
Sep 28 20:28:27 dist-test-slave-k477 krb5kdc[9824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:27.178452 10199 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:28:27.179014 10199 webserver.cc:460] Webserver started at http://127.24.186.195:39211/ using document root <none> and password file <none>
I20260928 20:28:27.179236 10199 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:27.179273 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:27.179363 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:27.181051 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "b036f760c2894cb98e883e561df27b00"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "46805b45e04e2b77a4201522c7d056d1"
server_key_iv: "f1dcfe31aa9c6b9a6352060ea7c85256"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.181593 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "b036f760c2894cb98e883e561df27b00"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "46805b45e04e2b77a4201522c7d056d1"
server_key_iv: "f1dcfe31aa9c6b9a6352060ea7c85256"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.183962 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 20:28:27.185050 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.185220 10199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:27.185328 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "b036f760c2894cb98e883e561df27b00"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "46805b45e04e2b77a4201522c7d056d1"
server_key_iv: "f1dcfe31aa9c6b9a6352060ea7c85256"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.185433 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:27.206627 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:27.208770 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:27.208936 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:27.209192 10199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:27.209524 10199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:27.209580 10199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.209621 10199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:27.209651 10199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.216869 10199 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:37753
I20260928 20:28:27.216918 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:37753 every 8 connection(s)
I20260928 20:28:27.217248 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:28:27 dist-test-slave-k477 krb5kdc[9824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627307, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:27.224931 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10199
I20260928 20:28:27.225020 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:27.225222 25323 external_mini_cluster.cc:1428] Setting key 6caa716fca64015d8e0a3f08edfa7cfb
I20260928 20:28:27.227210 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.217922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:44953 (local address 127.24.186.254:40455)
0928 20:28:27.217998 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.218000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.218199 (+ 199us) server_negotiation.cc:407] Connection header received
0928 20:28:27.218777 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.218778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.218788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.218851 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:27.219950 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.220457 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.221506 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.221763 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.221872 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.224267 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:27.224269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:27.224269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:27.224279 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:27.225983 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.226438 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.226439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.226439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.226496 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.226692 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.226692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.226692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.226805 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:27.226856 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.227073 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.227147 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 20:28:27.227416 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.217892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:40455 (local address 127.24.186.195:44953)
0928 20:28:27.218062 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.218103 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.218679 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.218925 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.218928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.219547 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:27.219856 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.219862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.220539 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.220540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.221351 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.221354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.221915 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.221916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.222030 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.222621 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:27.222632 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:27.224163 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:27.226074 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:27.226082 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:27.226085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:27.226367 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:27.226547 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:27.226548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:27.226548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:27.226638 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:27.226847 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:27.226850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:27.226965 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.227049 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.227175 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 20:28:27.227907 10328 heartbeater.cc:347] Connected to a master server at 127.24.186.254:40455
I20260928 20:28:27.228017 10328 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:27.228209 10328 heartbeater.cc:511] Master 127.24.186.254:40455 requested a full tablet report, sending...
I20260928 20:28:27.228588 9869 ts_manager.cc:194] Registered new tserver with Master: b036f760c2894cb98e883e561df27b00 (127.24.186.195:37753)
I20260928 20:28:27.229004 9869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:44953
I20260928 20:28:27.237632 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:27.246323 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.239093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.24.186.254:40455)
0928 20:28:27.239150 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.239152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.239178 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:28:27.239274 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.239275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.239286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.239358 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:27.239997 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.240496 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.241226 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.241423 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.241510 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.242747 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:27.242749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:27.242749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:27.242756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:27.245081 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.245443 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.245444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.245444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.245508 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:27.245706 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:27.245707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:27.245707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:27.245876 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:27.245997 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.246132 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.246219 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:28:27.248037 9869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:27.249715 9869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:27.257371 10340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.252749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35391 (local address 127.0.0.1:49264)
0928 20:28:27.252924 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.252966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.253038 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.253769 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.253770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.253771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:27.253813 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.253815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.254695 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.254696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.256033 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.256036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.257050 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.257051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.257138 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.257151 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.257240 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.257278 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":23,"thread_start_us":56,"threads_started":1}
I20260928 20:28:27.257388 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.252913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.24.186.193:35391)
0928 20:28:27.253542 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.253546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.253555 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:27.253595 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.253598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.253614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.253693 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:27.253887 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.254576 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256134 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.256879 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.257010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.257260 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.257304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":92,"threads_started":1}
I20260928 20:28:27.257453 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.253018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.24.186.194:34747)
0928 20:28:27.253202 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.253207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.253219 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:28:27.253263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.253265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.253289 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.253388 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:27.253698 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.254856 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256136 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.256879 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.257010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.257324 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.257363 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
I20260928 20:28:27.257756 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.252755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:34747 (local address 127.0.0.1:38450)
0928 20:28:27.253079 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.253088 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.253130 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.253581 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.253582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.253583 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:27.253617 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.253618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.254963 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.254964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.256041 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.256043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.257049 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.257050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.257138 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.257151 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.257219 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.257278 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":7,"thread_start_us":105,"threads_started":1}
I20260928 20:28:27.258184 9994 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a (DEFAULT_TABLE table=test-table [id=bf03923677c94d8a839114bfdd46dfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:27.258456 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.252755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:37753 (local address 127.0.0.1:43734)
0928 20:28:27.253146 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.253155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.253191 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.253416 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.253419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.253422 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:27.253497 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.253500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.256145 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.256846 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.256847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.258236 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.258237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.258284 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.258288 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.258329 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.258360 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":7,"thread_start_us":101,"threads_started":1}
I20260928 20:28:27.258477 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.253003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.24.186.195:37753)
0928 20:28:27.253203 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.253207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.253219 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:28:27.253263 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.253265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.253282 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.253388 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:27.253565 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.254852 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.256942 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.258138 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.258220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.258274 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.258354 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.258399 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260928 20:28:27.258575 9994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03c40bce154e6b95a5f77118b4b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:27.259152 10262 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a (DEFAULT_TABLE table=test-table [id=bf03923677c94d8a839114bfdd46dfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:27.259460 10262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03c40bce154e6b95a5f77118b4b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:27.262529 10345 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Bootstrap starting.
I20260928 20:28:27.262933 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a (DEFAULT_TABLE table=test-table [id=bf03923677c94d8a839114bfdd46dfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:27.263125 10346 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Bootstrap starting.
I20260928 20:28:27.263271 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f03c40bce154e6b95a5f77118b4b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:27.263345 10345 tablet_bootstrap.cc:654] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:27.263832 10346 tablet_bootstrap.cc:654] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:27.264006 10345 log.cc:824] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:27.264477 10346 log.cc:824] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:27.264811 10345 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: No bootstrap required, opened a new log
I20260928 20:28:27.264895 10345 ts_tablet_manager.cc:1397] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 20:28:27.265275 10346 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: No bootstrap required, opened a new log
I20260928 20:28:27.265345 10346 ts_tablet_manager.cc:1397] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:27.266321 10349 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Bootstrap starting.
I20260928 20:28:27.266888 10349 tablet_bootstrap.cc:654] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:27.266945 10345 raft_consensus.cc:348] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.267143 10345 raft_consensus.cc:374] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:27.267189 10345 raft_consensus.cc:729] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6a7b9b0103f49f5b482eee3a31f3323, State: Initialized, Role: FOLLOWER
I20260928 20:28:27.267302 10345 consensus_queue.cc:260] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.267328 10346 raft_consensus.cc:348] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.267465 10349 log.cc:824] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:27.267495 10346 raft_consensus.cc:374] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:27.267531 10346 raft_consensus.cc:729] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b036f760c2894cb98e883e561df27b00, State: Initialized, Role: FOLLOWER
I20260928 20:28:27.267580 10345 ts_tablet_manager.cc:1428] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:28:27.267645 10346 consensus_queue.cc:260] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.267884 10060 heartbeater.cc:503] Master 127.24.186.254:40455 was elected leader, sending a full tablet report...
I20260928 20:28:27.268074 10346 ts_tablet_manager.cc:1428] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:27.268229 10328 heartbeater.cc:503] Master 127.24.186.254:40455 was elected leader, sending a full tablet report...
I20260928 20:28:27.269107 10349 tablet_bootstrap.cc:492] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: No bootstrap required, opened a new log
I20260928 20:28:27.269213 10349 ts_tablet_manager.cc:1397] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:28:27.270653 10349 raft_consensus.cc:348] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.270787 10349 raft_consensus.cc:374] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:27.270807 10349 raft_consensus.cc:729] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077326239da94903b6a02026bd44feb3, State: Initialized, Role: FOLLOWER
I20260928 20:28:27.270915 10349 consensus_queue.cc:260] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.271153 10349 ts_tablet_manager.cc:1428] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:27.271210 10194 heartbeater.cc:503] Master 127.24.186.254:40455 was elected leader, sending a full tablet report...
W20260928 20:28:27.295409 10195 tablet.cc:2369] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:28:27.364046 10061 tablet.cc:2369] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:27.451653 10350 raft_consensus.cc:482] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:27.451790 10350 raft_consensus.cc:504] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.452198 10350 leader_election.cc:288] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b036f760c2894cb98e883e561df27b00 (127.24.186.195:37753), 077326239da94903b6a02026bd44feb3 (127.24.186.194:34747)
I20260928 20:28:27.456565 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.452492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:53509 (local address 127.24.186.194:34747)
0928 20:28:27.452541 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.452543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.452626 (+ 83us) server_negotiation.cc:407] Connection header received
0928 20:28:27.452686 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.452687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.452697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.452777 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:27.453005 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.453757 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.455361 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.455997 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.456089 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.456098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.456442 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.456492 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:28:27.456565 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.452489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:34493 (local address 127.24.186.195:37753)
0928 20:28:27.452541 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.452543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.452630 (+ 87us) server_negotiation.cc:407] Connection header received
0928 20:28:27.452706 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.452707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.452715 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.452777 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:27.453009 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.453774 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.455363 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.455997 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.456089 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.456098 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.456444 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.456492 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 20:28:27.456558 10354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.452400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:34747 (local address 127.24.186.193:53509)
0928 20:28:27.452537 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.452549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.452620 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.452798 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.452800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.452802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:27.452923 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.452926 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.453865 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.453866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.455237 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.455238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.456188 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.456188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.456288 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.456297 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.456369 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.456423 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":71,"thread_start_us":69,"threads_started":1}
I20260928 20:28:27.456557 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.452399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:37753 (local address 127.24.186.193:34493)
0928 20:28:27.452537 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.452549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.452620 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.452804 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.452804 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.452805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:27.452929 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.452930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.453865 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.453866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.455233 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.455236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.456186 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.456188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.456288 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.456295 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.456369 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.456423 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":115,"spinlock_wait_cycles":15744,"thread_start_us":77,"threads_started":1}
I20260928 20:28:27.456975 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03c40bce154e6b95a5f77118b4b20a" candidate_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036f760c2894cb98e883e561df27b00" is_pre_election: true
I20260928 20:28:27.457171 10282 raft_consensus.cc:2454] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a7b9b0103f49f5b482eee3a31f3323 in term 0.
I20260928 20:28:27.457140 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03c40bce154e6b95a5f77118b4b20a" candidate_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077326239da94903b6a02026bd44feb3" is_pre_election: true
I20260928 20:28:27.457300 10148 raft_consensus.cc:2454] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6a7b9b0103f49f5b482eee3a31f3323 in term 0.
I20260928 20:28:27.457661 9949 leader_election.cc:302] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b036f760c2894cb98e883e561df27b00, f6a7b9b0103f49f5b482eee3a31f3323; no voters:
I20260928 20:28:27.457860 10350 raft_consensus.cc:2781] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:27.457935 10350 raft_consensus.cc:482] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:27.457973 10350 raft_consensus.cc:3037] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:27.458732 10350 raft_consensus.cc:504] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.458895 10350 leader_election.cc:288] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [CANDIDATE]: Term 1 election: Requested vote from peers b036f760c2894cb98e883e561df27b00 (127.24.186.195:37753), 077326239da94903b6a02026bd44feb3 (127.24.186.194:34747)
I20260928 20:28:27.459067 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03c40bce154e6b95a5f77118b4b20a" candidate_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077326239da94903b6a02026bd44feb3"
I20260928 20:28:27.459082 10282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f03c40bce154e6b95a5f77118b4b20a" candidate_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b036f760c2894cb98e883e561df27b00"
I20260928 20:28:27.459163 10282 raft_consensus.cc:3037] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:27.459183 10148 raft_consensus.cc:3037] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:27.459993 10148 raft_consensus.cc:2454] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a7b9b0103f49f5b482eee3a31f3323 in term 1.
I20260928 20:28:27.460052 10282 raft_consensus.cc:2454] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6a7b9b0103f49f5b482eee3a31f3323 in term 1.
I20260928 20:28:27.460207 9950 leader_election.cc:302] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077326239da94903b6a02026bd44feb3, f6a7b9b0103f49f5b482eee3a31f3323; no voters:
I20260928 20:28:27.460342 10350 raft_consensus.cc:2781] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:27.460494 10350 raft_consensus.cc:686] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [term 1 LEADER]: Becoming Leader. State: Replica: f6a7b9b0103f49f5b482eee3a31f3323, State: Running, Role: LEADER
I20260928 20:28:27.460596 10350 consensus_queue.cc:237] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } }
I20260928 20:28:27.461505 9867 catalog_manager.cc:5495] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6a7b9b0103f49f5b482eee3a31f3323 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6a7b9b0103f49f5b482eee3a31f3323" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 } health_report { overall_health: UNKNOWN } } }
W20260928 20:28:27.467778 10329 tablet.cc:2369] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:27.485498 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.482362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.24.186.193:35391)
0928 20:28:27.482683 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.482686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.482706 (+ 20us) server_negotiation.cc:407] Connection header received
0928 20:28:27.482778 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.482780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.482810 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.482931 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:27.483163 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.483678 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.484614 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.484818 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.484886 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.485073 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:27.485187 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:27.485271 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.485351 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.485374 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260928 20:28:27.487439 10148 raft_consensus.cc:1261] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 1 FOLLOWER]: Refusing update from remote peer f6a7b9b0103f49f5b482eee3a31f3323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:27.487483 10282 raft_consensus.cc:1261] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 1 FOLLOWER]: Refusing update from remote peer f6a7b9b0103f49f5b482eee3a31f3323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:27.487876 10350 consensus_queue.cc:1035] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077326239da94903b6a02026bd44feb3" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:27.488047 10356 consensus_queue.cc:1035] T 4f03c40bce154e6b95a5f77118b4b20a P f6a7b9b0103f49f5b482eee3a31f3323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b036f760c2894cb98e883e561df27b00" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:27.491518 10360 mvcc.cc:205] Tried to move back new op lower bound from 7334409451465654272 to 7334409451359367168. Current Snapshot: MvccSnapshot[applied={T|T < 7334409451465654272 or (T in {7334409451465654272})}]
I20260928 20:28:27.491667 10361 mvcc.cc:205] Tried to move back new op lower bound from 7334409451465654272 to 7334409451359367168. Current Snapshot: MvccSnapshot[applied={T|T < 7334409451465654272}]
I20260928 20:28:27.493101 9869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:27.493224 9869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:27.493990 9869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d13fec58601d4dd38d74fcafadbe59fa: Sending DeleteTablet for 3 replicas of tablet 4f03c40bce154e6b95a5f77118b4b20a
I20260928 20:28:27.494333 10262 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:27 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:43734
I20260928 20:28:27.494360 9994 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:27 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:49264
I20260928 20:28:27.494359 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f03c40bce154e6b95a5f77118b4b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:27 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:38450
I20260928 20:28:27.494582 10370 tablet_replica.cc:330] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: stopping tablet replica
I20260928 20:28:27.494657 10368 tablet_replica.cc:330] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: stopping tablet replica
I20260928 20:28:27.494688 10370 raft_consensus.cc:2229] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:27.494740 10368 raft_consensus.cc:2229] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:27.494796 10370 raft_consensus.cc:2258] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:27.494820 10368 raft_consensus.cc:2258] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:27.495106 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9931
I20260928 20:28:27.495414 10370 ts_tablet_manager.cc:1905] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:27.495484 10368 ts_tablet_manager.cc:1905] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:27.497575 10368 ts_tablet_manager.cc:1918] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:27.497651 10368 log.cc:1196] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/4f03c40bce154e6b95a5f77118b4b20a
I20260928 20:28:27.497941 10368 ts_tablet_manager.cc:1939] T 4f03c40bce154e6b95a5f77118b4b20a P b036f760c2894cb98e883e561df27b00: Deleting consensus metadata
I20260928 20:28:27.498358 10370 ts_tablet_manager.cc:1918] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:27.498440 10370 log.cc:1196] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/4f03c40bce154e6b95a5f77118b4b20a
I20260928 20:28:27.498399 9855 catalog_manager.cc:4841] TS b036f760c2894cb98e883e561df27b00 (127.24.186.195:37753): tablet 4f03c40bce154e6b95a5f77118b4b20a (table test-table [id=bf03923677c94d8a839114bfdd46dfd8]) successfully deleted
I20260928 20:28:27.498713 10370 ts_tablet_manager.cc:1939] T 4f03c40bce154e6b95a5f77118b4b20a P 077326239da94903b6a02026bd44feb3: Deleting consensus metadata
I20260928 20:28:27.499123 9856 catalog_manager.cc:4841] TS 077326239da94903b6a02026bd44feb3 (127.24.186.194:34747): tablet 4f03c40bce154e6b95a5f77118b4b20a (table test-table [id=bf03923677c94d8a839114bfdd46dfd8]) successfully deleted
W20260928 20:28:27.501746 9853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:35391 (error 108)
I20260928 20:28:27.501978 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10065
I20260928 20:28:27.502559 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.502161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:35391 (local address 127.0.0.1:49286)
0928 20:28:27.502487 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.502518 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307}
W20260928 20:28:27.502774 9853 catalog_manager.cc:4584] TS f6a7b9b0103f49f5b482eee3a31f3323 (127.24.186.193:35391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f03c40bce154e6b95a5f77118b4b20a: Network error: Client connection negotiation failed: client connection to 127.24.186.193:35391: connect: Connection refused (error 111)
I20260928 20:28:27.507165 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10199
I20260928 20:28:27.512251 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 9837
2026-09-28T20:28:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2188 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T20:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:27Z Disabled control of system clock
I20260928 20:28:27.543236 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35791
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:27.631189 10375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:27.631510 10375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:27.631548 10375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:27.632993 10375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:27.633082 10375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:27.633105 10375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:27.633134 10375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:27.633174 10375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:27.635581 10375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:27.635970 10375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:27.636441 10375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:27.641580 10383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.641664 10381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.641611 10380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:27.641855 10375 server_base.cc:1034] running on GCE node
I20260928 20:28:27.642089 10375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:27.642402 10375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:27.643571 10375 hybrid_clock.cc:648] HybridClock initialized: now 1790627307643548 us; error 42 us; skew 500 ppm
I20260928 20:28:27.644975 10375 webserver.cc:460] Webserver started at http://127.24.186.254:33063/ using document root <none> and password file <none>
I20260928 20:28:27.645274 10375 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:27.645335 10375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:27.645478 10375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:27.646616 10375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "fbf214bf88dd443c9d4e86f42c6e24db"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "230d17c895cfe2a6e522d0e4352a92e1"
server_key_iv: "689d11d544a9e0e5496af255908e2cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.646982 10375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "fbf214bf88dd443c9d4e86f42c6e24db"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "230d17c895cfe2a6e522d0e4352a92e1"
server_key_iv: "689d11d544a9e0e5496af255908e2cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.648452 10375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:28:27.649261 10389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.649461 10375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:28:27.649570 10375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "fbf214bf88dd443c9d4e86f42c6e24db"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "230d17c895cfe2a6e522d0e4352a92e1"
server_key_iv: "689d11d544a9e0e5496af255908e2cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.649678 10375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:27.681121 10375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:27.681437 10375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:27.681546 10375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:27.686273 10375 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:35791
I20260928 20:28:27.686343 10441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:35791 every 8 connection(s)
I20260928 20:28:27.686691 10375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:27.687723 10442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:27.689095 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10375
I20260928 20:28:27.689181 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:27.689391 25323 external_mini_cluster.cc:1428] Setting key 09273de2bfe5c88ccf08face1f00b8cb
I20260928 20:28:27.690429 10442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: Bootstrap starting.
I20260928 20:28:27.691139 10442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:27.691445 10442 log.cc:824] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:27.692544 10442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: No bootstrap required, opened a new log
I20260928 20:28:27.694778 10442 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } }
I20260928 20:28:27.694983 10442 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:27.695032 10442 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf214bf88dd443c9d4e86f42c6e24db, State: Initialized, Role: FOLLOWER
I20260928 20:28:27.695176 10442 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } }
I20260928 20:28:27.695276 10442 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:27.695307 10442 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:27.695364 10442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:27.696202 10442 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } }
I20260928 20:28:27.696318 10442 leader_election.cc:302] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbf214bf88dd443c9d4e86f42c6e24db; no voters:
I20260928 20:28:27.696547 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.689969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.24.186.254:35791)
0928 20:28:27.690159 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.690166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.690190 (+ 24us) server_negotiation.cc:407] Connection header received
0928 20:28:27.690624 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.690631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.690691 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.690803 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:27.691096 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.693140 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.694337 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.694788 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.694893 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.695727 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:27.695731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:27.695731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:27.695962 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:27.695995 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:27.696006 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:27.696058 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.696160 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.696230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260928 20:28:27.696586 10447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:27.696549 10442 leader_election.cc:288] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:27.696753 10447 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [term 1 LEADER]: Becoming Leader. State: Replica: fbf214bf88dd443c9d4e86f42c6e24db, State: Running, Role: LEADER
I20260928 20:28:27.696834 10447 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } }
I20260928 20:28:27.696867 10442 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:27.697134 10447 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } } }
I20260928 20:28:27.697198 10447 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:27.697203 10448 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbf214bf88dd443c9d4e86f42c6e24db. Latest consensus state: current_term: 1 leader_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbf214bf88dd443c9d4e86f42c6e24db" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35791 } } }
I20260928 20:28:27.697304 10448 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:27.698728 10463 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:28:27.698774 10463 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:27.698837 10452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:27.699013 10452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:27.700425 10452 catalog_manager.cc:1347] Generated new cluster ID: affd7fe9d25e4448a656af2cfd97791a
I20260928 20:28:27.700481 10452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:27.732833 10452 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:27.733990 10452 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:27.736990 10452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: Generated new TSK 0
I20260928 20:28:27.737207 10452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:27.749706 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--builtin_ntp_servers=127.24.186.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:27.835911 10466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:27.836133 10466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:27.836158 10466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:27.837473 10466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:27.837519 10466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:27.837546 10466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:27.838938 10466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:27.839207 10466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:27.839522 10466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:27.844596 10472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.844599 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.844700 10474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:27.845508 10466 server_base.cc:1034] running on GCE node
I20260928 20:28:27.845891 10466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:27.846505 10466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:27.847759 10466 hybrid_clock.cc:648] HybridClock initialized: now 1790627307847733 us; error 51 us; skew 500 ppm
I20260928 20:28:27.849284 10466 webserver.cc:460] Webserver started at http://127.24.186.193:36261/ using document root <none> and password file <none>
I20260928 20:28:27.849543 10466 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:27.849613 10466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:27.849766 10466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:27.850869 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "cae55d5b6e7a48e3b90f7a4e5db43842"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "fdc9dcae60621497d5c0586d7e5dc391"
server_key_iv: "c9c6b4c5277fb45dce449bb7f92634c5"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.851234 10466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "cae55d5b6e7a48e3b90f7a4e5db43842"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "fdc9dcae60621497d5c0586d7e5dc391"
server_key_iv: "c9c6b4c5277fb45dce449bb7f92634c5"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.852728 10466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:27.853538 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.853736 10466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:27.853847 10466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "cae55d5b6e7a48e3b90f7a4e5db43842"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "fdc9dcae60621497d5c0586d7e5dc391"
server_key_iv: "c9c6b4c5277fb45dce449bb7f92634c5"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.853928 10466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:27.867868 10466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:27.868223 10466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:27.868384 10466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:27.868626 10466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:27.868968 10466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:27.869019 10466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.869060 10466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:27.869089 10466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.875241 10466 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:45933
I20260928 20:28:27.875303 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:45933 every 8 connection(s)
I20260928 20:28:27.875641 10466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:27.881307 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.876643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:37165 (local address 127.24.186.254:35791)
0928 20:28:27.876727 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:27.876729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:27.877008 (+ 279us) server_negotiation.cc:407] Connection header received
0928 20:28:27.877577 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:27.877579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:27.877596 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:27.877645 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:27.878337 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.878850 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.879857 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.880070 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.880148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.880995 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:27.880997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:27.880997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:27.881002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:27.881017 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:27.881021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:27.881087 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:27.881192 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 20:28:27.881219 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:28:27.881574 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:27.876625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35791 (local address 127.24.186.193:37165)
0928 20:28:27.876882 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:27.876916 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:27.877462 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:27.877739 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:27.877742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:27.877750 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:27.878236 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.878239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.878944 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.878945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.879738 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:27.879741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:27.880156 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:27.880157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:27.880296 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:27.880852 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:27.880853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:27.880881 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:27.880902 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:27.881088 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:27.881095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:27.881098 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:27.881187 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 20:28:27.881307 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":145,"threads_started":1}
I20260928 20:28:27.882088 10594 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35791
I20260928 20:28:27.882182 10594 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:27.882334 10594 heartbeater.cc:511] Master 127.24.186.254:35791 requested a full tablet report, sending...
I20260928 20:28:27.882807 10406 ts_manager.cc:194] Registered new tserver with Master: cae55d5b6e7a48e3b90f7a4e5db43842 (127.24.186.193:45933)
I20260928 20:28:27.883711 10406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.193:37165
I20260928 20:28:27.885277 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10466
I20260928 20:28:27.885511 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:27.885762 25323 external_mini_cluster.cc:1428] Setting key d7e3f6844a483ebdffea72475477e9bb
I20260928 20:28:27.886854 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--builtin_ntp_servers=127.24.186.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:27.975871 10597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:27.976099 10597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:27.976125 10597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:27.977487 10597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:27.977553 10597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:27.977584 10597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:27.979221 10597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:27.979451 10597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:27.979825 10597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:28:27.985802 10597 server_base.cc:1034] running on GCE node
W20260928 20:28:27.985741 10602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.985764 10605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:27.985769 10603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:27.986407 10597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:27.986754 10597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:27.987897 10597 hybrid_clock.cc:648] HybridClock initialized: now 1790627307987873 us; error 41 us; skew 500 ppm
I20260928 20:28:27.989254 10597 webserver.cc:460] Webserver started at http://127.24.186.194:33433/ using document root <none> and password file <none>
I20260928 20:28:27.989533 10597 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:27.989606 10597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:27.989771 10597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:27.991950 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "c79d2cea12174f1db7a14ae276151d07"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "d09010a7cdc58a67e9a80bfb8184a0e7"
server_key_iv: "9dcefbbd857a56dc803844654dec32b9"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.992529 10597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "c79d2cea12174f1db7a14ae276151d07"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "d09010a7cdc58a67e9a80bfb8184a0e7"
server_key_iv: "9dcefbbd857a56dc803844654dec32b9"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.994469 10597 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:27.995425 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:27.995643 10597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:27.995767 10597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "c79d2cea12174f1db7a14ae276151d07"
format_stamp: "Formatted at 2026-09-28 20:28:27 on dist-test-slave-k477"
server_key: "d09010a7cdc58a67e9a80bfb8184a0e7"
server_key_iv: "9dcefbbd857a56dc803844654dec32b9"
server_key_version: "encryptionkey@0"
I20260928 20:28:27.995918 10597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:28.016825 10597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:28.017170 10597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:28.017338 10597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:28.017587 10597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:28.017954 10597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:28.018018 10597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:28.018060 10597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:28.018090 10597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:28.025170 10597 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:33667
I20260928 20:28:28.025224 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:33667 every 8 connection(s)
I20260928 20:28:28.025549 10597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:28.030982 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.026329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:38661 (local address 127.24.186.254:35791)
0928 20:28:28.026418 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.026422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.026632 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:28:28.027139 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.027140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.027151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.027219 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:28.027930 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.028573 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.029566 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.029759 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.029824 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.030748 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:28.030749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:28.030749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:28.030753 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:28.030768 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:28.030771 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:28.030818 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.030896 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.030917 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 20:28:28.031239 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.026301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35791 (local address 127.24.186.194:38661)
0928 20:28:28.026497 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.026540 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.027043 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.027296 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.027299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.027308 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:28.027850 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.027854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.028691 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.028692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.029478 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.029479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.029869 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.029870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.029998 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.030615 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:28.030615 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:28.030644 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:28.030666 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:28.030829 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:28.030832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:28.030835 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.030887 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.030970 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 20:28:28.031764 10725 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35791
I20260928 20:28:28.031914 10725 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:28.032120 10725 heartbeater.cc:511] Master 127.24.186.254:35791 requested a full tablet report, sending...
I20260928 20:28:28.032601 10406 ts_manager.cc:194] Registered new tserver with Master: c79d2cea12174f1db7a14ae276151d07 (127.24.186.194:33667)
I20260928 20:28:28.032915 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10597
I20260928 20:28:28.033041 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:28.033083 10406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.194:38661
I20260928 20:28:28.033300 25323 external_mini_cluster.cc:1428] Setting key faba3a8de7efa04dc38221d1abae8acd
I20260928 20:28:28.035143 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--builtin_ntp_servers=127.24.186.212:36757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:28.128733 10728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:28.128965 10728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:28.128988 10728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:28.130214 10728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:28.130270 10728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:28.130300 10728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:28.131740 10728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:35791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:28.132078 10728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:28.132416 10728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:28.137923 10734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:28.137879 10733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:28.137876 10736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:28.138226 10728 server_base.cc:1034] running on GCE node
I20260928 20:28:28.138520 10728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:28.138854 10728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:28.140007 10728 hybrid_clock.cc:648] HybridClock initialized: now 1790627308139987 us; error 39 us; skew 500 ppm
I20260928 20:28:28.141439 10728 webserver.cc:460] Webserver started at http://127.24.186.195:39157/ using document root <none> and password file <none>
I20260928 20:28:28.141721 10728 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:28.141772 10728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:28.141938 10728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:28.143035 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "6fe1add5af9743b0ab8a5e059d2e277c"
format_stamp: "Formatted at 2026-09-28 20:28:28 on dist-test-slave-k477"
server_key: "986f1ce8bffe4558a13a25a302817b6c"
server_key_iv: "d30a933d799778aa47cba90fe43feafe"
server_key_version: "encryptionkey@0"
I20260928 20:28:28.143400 10728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "6fe1add5af9743b0ab8a5e059d2e277c"
format_stamp: "Formatted at 2026-09-28 20:28:28 on dist-test-slave-k477"
server_key: "986f1ce8bffe4558a13a25a302817b6c"
server_key_iv: "d30a933d799778aa47cba90fe43feafe"
server_key_version: "encryptionkey@0"
I20260928 20:28:28.144912 10728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:28.146034 10742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:28.146328 10728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:28.146448 10728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "6fe1add5af9743b0ab8a5e059d2e277c"
format_stamp: "Formatted at 2026-09-28 20:28:28 on dist-test-slave-k477"
server_key: "986f1ce8bffe4558a13a25a302817b6c"
server_key_iv: "d30a933d799778aa47cba90fe43feafe"
server_key_version: "encryptionkey@0"
I20260928 20:28:28.146522 10728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:28.165314 10728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:28.165614 10728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:28.165726 10728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:28.165932 10728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:28.166292 10728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:28.166327 10728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:28.166384 10728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:28.166411 10728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:28.172358 10728 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:42473
I20260928 20:28:28.172459 10855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:42473 every 8 connection(s)
I20260928 20:28:28.172758 10728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:28:28.178218 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.173581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:60931 (local address 127.24.186.254:35791)
0928 20:28:28.173627 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.173629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.173890 (+ 261us) server_negotiation.cc:407] Connection header received
0928 20:28:28.174432 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.174433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.174446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.174515 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:28.175217 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.175717 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.176751 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.176944 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.177034 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.177902 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:28.177903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:28.177904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:28.177909 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:28.177929 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:28.177933 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:28.177997 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.178092 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.178131 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:28:28.178443 10858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.173566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35791 (local address 127.24.186.195:60931)
0928 20:28:28.173770 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.173806 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.174325 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.174594 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.174600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.174614 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:28.175111 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.175115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.175856 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.175857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.176653 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.176655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.177044 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.177046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.177184 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.177731 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:28.177732 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:28.177760 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:28.177783 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:28.177997 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:28.178004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:28.178007 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.178094 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.178199 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260928 20:28:28.179023 10856 heartbeater.cc:347] Connected to a master server at 127.24.186.254:35791
I20260928 20:28:28.179157 10856 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:28.179347 10856 heartbeater.cc:511] Master 127.24.186.254:35791 requested a full tablet report, sending...
I20260928 20:28:28.179736 10406 ts_manager.cc:194] Registered new tserver with Master: 6fe1add5af9743b0ab8a5e059d2e277c (127.24.186.195:42473)
I20260928 20:28:28.180146 10406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.195:60931
I20260928 20:28:28.181318 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10728
I20260928 20:28:28.181419 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:28.181677 25323 external_mini_cluster.cc:1428] Setting key b24536c295d46f728b100f8928ab5146
I20260928 20:28:28.184011 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:28.188270 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.185439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.24.186.254:35791)
0928 20:28:28.185521 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.185523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.185554 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:28:28.185620 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.185621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.185631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.185680 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:28.185826 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.186327 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.187080 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.187264 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.187317 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.187993 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:28.187995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:28.187995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:28.187999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:28.188012 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:28.188016 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:28.188059 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.188132 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.188152 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 20:28:28.189920 10406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:28.191464 10406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:28.198642 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.193818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:33667 (local address 127.0.0.1:37234)
0928 20:28:28.193942 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.193981 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.194050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.194690 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.194691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.194693 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:28.194746 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.194749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.196107 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.196108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.197104 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.197106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.198392 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.198393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.198449 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.198452 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.198505 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.198547 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260928 20:28:28.198655 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.194074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55254 (local address 127.24.186.193:45933)
0928 20:28:28.194301 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.194311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.194320 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:28.194358 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.194359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.194371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.194454 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:28.194643 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.195444 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.196972 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.197995 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.198078 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.198182 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.198526 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.198570 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":121,"threads_started":1}
I20260928 20:28:28.198693 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.194277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.24.186.194:33667)
0928 20:28:28.194440 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.194445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.194457 (+ 12us) server_negotiation.cc:407] Connection header received
0928 20:28:28.194522 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.194524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.194540 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.194622 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:28.194828 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.196015 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.197175 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.198212 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.198278 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.198359 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.198536 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.198587 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 20:28:28.198850 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.193820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45933 (local address 127.0.0.1:55254)
0928 20:28:28.194064 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.194075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.194130 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.194470 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.194473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.194475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:28.194546 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.194549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.195534 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.195536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.196893 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.196896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.198223 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.198224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.198314 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.198328 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.198404 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.198772 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":57,"threads_started":1}
I20260928 20:28:28.199683 10528 tablet_service.cc:1467] Processing CreateTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f (DEFAULT_TABLE table=test-table [id=ffd81ec1a16a4683b5a9d26f7457dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:28.199692 10659 tablet_service.cc:1467] Processing CreateTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f (DEFAULT_TABLE table=test-table [id=ffd81ec1a16a4683b5a9d26f7457dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:28.200125 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ec4cf42bba4dcd81ee6a49a58c745f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:28.200125 10528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ec4cf42bba4dcd81ee6a49a58c745f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:28.200999 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.193821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:42473 (local address 127.0.0.1:40206)
0928 20:28:28.194168 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.194177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.194216 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.194694 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.194695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.194696 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:28.194746 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.194748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.197175 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.197175 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.198561 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.198564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.200726 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.200727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.200797 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.200803 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.200866 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.200920 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":9,"thread_start_us":74,"threads_started":1}
I20260928 20:28:28.201009 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.194074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.24.186.195:42473)
0928 20:28:28.194483 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.194486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.194497 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:28:28.194533 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.194535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.194548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.194622 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:28.194829 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.196023 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.198669 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.200621 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.200727 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.200811 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.200880 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.200919 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":106,"threads_started":1}
I20260928 20:28:28.201886 10790 tablet_service.cc:1467] Processing CreateTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f (DEFAULT_TABLE table=test-table [id=ffd81ec1a16a4683b5a9d26f7457dead]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:28.202214 10790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9ec4cf42bba4dcd81ee6a49a58c745f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:28.203653 10870 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: Bootstrap starting.
I20260928 20:28:28.204456 10870 tablet_bootstrap.cc:654] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:28.204795 10870 log.cc:824] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:28.205145 10871 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Bootstrap starting.
I20260928 20:28:28.205590 10873 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Bootstrap starting.
I20260928 20:28:28.205708 10870 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: No bootstrap required, opened a new log
I20260928 20:28:28.205780 10870 ts_tablet_manager.cc:1397] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:28:28.205919 10871 tablet_bootstrap.cc:654] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:28.206216 10871 log.cc:824] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:28.206362 10873 tablet_bootstrap.cc:654] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:28.206926 10873 log.cc:824] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:28.207684 10873 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: No bootstrap required, opened a new log
I20260928 20:28:28.207753 10873 ts_tablet_manager.cc:1397] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 20:28:28.208317 10871 tablet_bootstrap.cc:492] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: No bootstrap required, opened a new log
I20260928 20:28:28.208396 10871 ts_tablet_manager.cc:1397] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:28:28.208762 10870 raft_consensus.cc:348] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.208936 10870 raft_consensus.cc:374] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:28.208971 10870 raft_consensus.cc:729] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae55d5b6e7a48e3b90f7a4e5db43842, State: Initialized, Role: FOLLOWER
I20260928 20:28:28.209115 10870 consensus_queue.cc:260] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.209408 10870 ts_tablet_manager.cc:1428] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 20:28:28.209524 10594 heartbeater.cc:503] Master 127.24.186.254:35791 was elected leader, sending a full tablet report...
I20260928 20:28:28.210455 10873 raft_consensus.cc:348] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.210654 10873 raft_consensus.cc:374] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:28.210700 10873 raft_consensus.cc:729] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fe1add5af9743b0ab8a5e059d2e277c, State: Initialized, Role: FOLLOWER
I20260928 20:28:28.210826 10873 consensus_queue.cc:260] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.211007 10871 raft_consensus.cc:348] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.211114 10873 ts_tablet_manager.cc:1428] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 20:28:28.211144 10856 heartbeater.cc:503] Master 127.24.186.254:35791 was elected leader, sending a full tablet report...
I20260928 20:28:28.211158 10871 raft_consensus.cc:374] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:28.211191 10871 raft_consensus.cc:729] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79d2cea12174f1db7a14ae276151d07, State: Initialized, Role: FOLLOWER
I20260928 20:28:28.211295 10871 consensus_queue.cc:260] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.211544 10871 ts_tablet_manager.cc:1428] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:28.211594 10725 heartbeater.cc:503] Master 127.24.186.254:35791 was elected leader, sending a full tablet report...
I20260928 20:28:28.215651 10878 raft_consensus.cc:482] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:28.215758 10878 raft_consensus.cc:504] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.216158 10878 leader_election.cc:288] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fe1add5af9743b0ab8a5e059d2e277c (127.24.186.195:42473), cae55d5b6e7a48e3b90f7a4e5db43842 (127.24.186.193:45933)
I20260928 20:28:28.219120 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.216294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:42473 (local address 127.24.186.194:53621)
0928 20:28:28.216332 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.216345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.216390 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.216843 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.216844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.216846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:28.216916 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.216918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.217540 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.217541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.218298 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.218300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.218864 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.218865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.218940 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.218943 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.219008 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.219051 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":5}
I20260928 20:28:28.219120 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.216581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:53621 (local address 127.24.186.195:42473)
0928 20:28:28.216669 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.216672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.216682 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:28:28.216712 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.216713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.216723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.216787 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:28.216980 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.217456 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.218394 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.218771 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.218864 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.218872 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.219013 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.219051 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 20:28:28.219333 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.216612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45933 (local address 127.24.186.194:59361)
0928 20:28:28.216797 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.216809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:28.216864 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:28.217029 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:28.217030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:28.217032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:28.217107 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.217108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.217761 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.217761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.218493 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:28.218495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.219038 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.219039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:28.219107 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.219111 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:28.219184 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:28:28.219235 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":130,"threads_started":1}
I20260928 20:28:28.219430 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.216739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:59361 (local address 127.24.186.193:45933)
0928 20:28:28.216788 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.216790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.216860 (+ 70us) server_negotiation.cc:407] Connection header received
0928 20:28:28.216923 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.216924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.216932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.217021 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:28.217190 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.217668 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.218580 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.218943 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.219036 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.219043 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.219309 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.219354 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:28:28.219455 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ec4cf42bba4dcd81ee6a49a58c745f" candidate_uuid: "c79d2cea12174f1db7a14ae276151d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" is_pre_election: true
I20260928 20:28:28.219620 10810 raft_consensus.cc:2454] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79d2cea12174f1db7a14ae276151d07 in term 0.
I20260928 20:28:28.219681 10548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ec4cf42bba4dcd81ee6a49a58c745f" candidate_uuid: "c79d2cea12174f1db7a14ae276151d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" is_pre_election: true
I20260928 20:28:28.219851 10548 raft_consensus.cc:2454] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79d2cea12174f1db7a14ae276151d07 in term 0.
I20260928 20:28:28.219880 10612 leader_election.cc:302] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe1add5af9743b0ab8a5e059d2e277c, c79d2cea12174f1db7a14ae276151d07; no voters:
I20260928 20:28:28.219996 10878 raft_consensus.cc:2781] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:28.220048 10878 raft_consensus.cc:482] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:28.220075 10878 raft_consensus.cc:3037] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:28.221026 10878 raft_consensus.cc:504] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.221169 10878 leader_election.cc:288] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [CANDIDATE]: Term 1 election: Requested vote from peers 6fe1add5af9743b0ab8a5e059d2e277c (127.24.186.195:42473), cae55d5b6e7a48e3b90f7a4e5db43842 (127.24.186.193:45933)
I20260928 20:28:28.221297 10810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ec4cf42bba4dcd81ee6a49a58c745f" candidate_uuid: "c79d2cea12174f1db7a14ae276151d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fe1add5af9743b0ab8a5e059d2e277c"
I20260928 20:28:28.221377 10810 raft_consensus.cc:3037] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:28.221344 10548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9ec4cf42bba4dcd81ee6a49a58c745f" candidate_uuid: "c79d2cea12174f1db7a14ae276151d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842"
I20260928 20:28:28.221457 10548 raft_consensus.cc:3037] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:28.222036 10810 raft_consensus.cc:2454] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79d2cea12174f1db7a14ae276151d07 in term 1.
I20260928 20:28:28.222199 10612 leader_election.cc:302] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fe1add5af9743b0ab8a5e059d2e277c, c79d2cea12174f1db7a14ae276151d07; no voters:
I20260928 20:28:28.222298 10878 raft_consensus.cc:2781] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:28.222319 10548 raft_consensus.cc:2454] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79d2cea12174f1db7a14ae276151d07 in term 1.
I20260928 20:28:28.222466 10878 raft_consensus.cc:686] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 1 LEADER]: Becoming Leader. State: Replica: c79d2cea12174f1db7a14ae276151d07, State: Running, Role: LEADER
I20260928 20:28:28.222560 10878 consensus_queue.cc:237] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } }
I20260928 20:28:28.223407 10405 catalog_manager.cc:5495] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to c79d2cea12174f1db7a14ae276151d07 (127.24.186.194). New cstate: current_term: 1 leader_uuid: "c79d2cea12174f1db7a14ae276151d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c79d2cea12174f1db7a14ae276151d07" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 33667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 } health_report { overall_health: UNKNOWN } } }
I20260928 20:28:28.231251 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.229134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.24.186.194:33667)
0928 20:28:28.229173 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:28.229175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:28.229181 (+ 6us) server_negotiation.cc:407] Connection header received
0928 20:28:28.229211 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:28.229212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:28.229234 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:28.229307 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:28.229460 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.229916 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.230612 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:28.230800 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:28.230858 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:28.230965 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:28.231061 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:28.231117 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:28.231146 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 20:28:28.231163 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:28:28.233400 10548 raft_consensus.cc:1261] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 1 FOLLOWER]: Refusing update from remote peer c79d2cea12174f1db7a14ae276151d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:28.233417 10810 raft_consensus.cc:1261] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 1 FOLLOWER]: Refusing update from remote peer c79d2cea12174f1db7a14ae276151d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:28.233747 10878 consensus_queue.cc:1035] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fe1add5af9743b0ab8a5e059d2e277c" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 42473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:28.233886 10880 consensus_queue.cc:1035] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae55d5b6e7a48e3b90f7a4e5db43842" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:28.238221 10884 mvcc.cc:205] Tried to move back new op lower bound from 7334409454520750080 to 7334409454480392192. Current Snapshot: MvccSnapshot[applied={T|T < 7334409454520750080}]
I20260928 20:28:28.238360 10885 mvcc.cc:205] Tried to move back new op lower bound from 7334409454520750080 to 7334409454480392192. Current Snapshot: MvccSnapshot[applied={T|T < 7334409454520750080 or (T in {7334409454520750080})}]
I20260928 20:28:28.238363 10883 mvcc.cc:205] Tried to move back new op lower bound from 7334409454520750080 to 7334409454480392192. Current Snapshot: MvccSnapshot[applied={T|T < 7334409454520750080}]
I20260928 20:28:28.240662 10404 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:28.240782 10404 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:28.241636 10404 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbf214bf88dd443c9d4e86f42c6e24db: Sending DeleteTablet for 3 replicas of tablet a9ec4cf42bba4dcd81ee6a49a58c745f
I20260928 20:28:28.241940 10659 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:28 UTC) from {username='slave'} at 127.0.0.1:37234
I20260928 20:28:28.241978 10528 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:28 UTC) from {username='slave'} at 127.0.0.1:55254
I20260928 20:28:28.242038 10790 tablet_service.cc:1514] Processing DeleteTablet for tablet a9ec4cf42bba4dcd81ee6a49a58c745f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:28 UTC) from {username='slave'} at 127.0.0.1:40206
I20260928 20:28:28.242283 10895 tablet_replica.cc:330] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: stopping tablet replica
I20260928 20:28:28.242285 10894 tablet_replica.cc:330] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842: stopping tablet replica
I20260928 20:28:28.242390 10895 raft_consensus.cc:2229] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:28.242391 10894 raft_consensus.cc:2229] T a9ec4cf42bba4dcd81ee6a49a58c745f P cae55d5b6e7a48e3b90f7a4e5db43842 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:28.242552 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10466
I20260928 20:28:28.243199 10893 tablet_replica.cc:330] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: stopping tablet replica
I20260928 20:28:28.243294 10893 raft_consensus.cc:2229] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:28:28.243413 10893 raft_consensus.cc:2258] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:28.245088 10893 ts_tablet_manager.cc:1905] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:28.246690 10893 ts_tablet_manager.cc:1918] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:28.246800 10893 log.cc:1196] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/a9ec4cf42bba4dcd81ee6a49a58c745f
I20260928 20:28:28.247037 10893 ts_tablet_manager.cc:1939] T a9ec4cf42bba4dcd81ee6a49a58c745f P c79d2cea12174f1db7a14ae276151d07: Deleting consensus metadata
I20260928 20:28:28.247475 10392 catalog_manager.cc:4841] TS c79d2cea12174f1db7a14ae276151d07 (127.24.186.194:33667): tablet a9ec4cf42bba4dcd81ee6a49a58c745f (table test-table [id=ffd81ec1a16a4683b5a9d26f7457dead]) successfully deleted
I20260928 20:28:28.242506 10895 raft_consensus.cc:2258] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:28.248342 10895 ts_tablet_manager.cc:1905] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 20:28:28.249323 10391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:45933 (error 108)
I20260928 20:28:28.249476 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10597
I20260928 20:28:28.250077 10895 ts_tablet_manager.cc:1918] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:28.250152 10895 log.cc:1196] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/a9ec4cf42bba4dcd81ee6a49a58c745f
I20260928 20:28:28.250403 10895 ts_tablet_manager.cc:1939] T a9ec4cf42bba4dcd81ee6a49a58c745f P 6fe1add5af9743b0ab8a5e059d2e277c: Deleting consensus metadata
I20260928 20:28:28.250911 10390 catalog_manager.cc:4841] TS 6fe1add5af9743b0ab8a5e059d2e277c (127.24.186.195:42473): tablet a9ec4cf42bba4dcd81ee6a49a58c745f (table test-table [id=ffd81ec1a16a4683b5a9d26f7457dead]) successfully deleted
I20260928 20:28:28.251145 10864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:28.251029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:45933 (local address 127.0.0.1:55262)
0928 20:28:28.251069 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:28.251115 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:45933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 20:28:28.251282 10391 catalog_manager.cc:4584] TS cae55d5b6e7a48e3b90f7a4e5db43842 (127.24.186.193:45933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9ec4cf42bba4dcd81ee6a49a58c745f: Network error: Client connection negotiation failed: client connection to 127.24.186.193:45933: connect: Connection refused (error 111)
I20260928 20:28:28.255122 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10728
I20260928 20:28:28.260667 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10375
2026-09-28T20:28:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (746 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:28 dist-test-slave-k477 krb5kdc[10899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:28 dist-test-slave-k477 krb5kdc[10899](info): set up 2 sockets
Sep 28 20:28:28 dist-test-slave-k477 krb5kdc[10899](info): commencing operation
krb5kdc: starting...
W20260928 20:28:29.329169 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:29.463951 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:36861
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:29.556185 10912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:29.556447 10912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:29.556471 10912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:29.557768 10912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:29.557818 10912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:29.557833 10912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:29.557847 10912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:29.557859 10912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:29.559397 10912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:36861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:29.559623 10912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:29.559840 10912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:29.566251 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.566354 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.566265 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:29.566669 10912 server_base.cc:1034] running on GCE node
I20260928 20:28:29.566951 10912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:29.567266 10912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:29.568421 10912 hybrid_clock.cc:648] HybridClock initialized: now 1790627309568405 us; error 48 us; skew 500 ppm
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:29.571110 10912 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:29.571640 10912 webserver.cc:460] Webserver started at http://127.24.186.254:41693/ using document root <none> and password file <none>
I20260928 20:28:29.571944 10912 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:29.571995 10912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:29.572135 10912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:29.573483 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "16d0e63367c04226b2d004c1807ae126"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "790e44aabad30dd61e0614a9e9df20a3"
server_key_iv: "372420e1b4dc9cce16329f404d3b35f1"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.573905 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "16d0e63367c04226b2d004c1807ae126"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "790e44aabad30dd61e0614a9e9df20a3"
server_key_iv: "372420e1b4dc9cce16329f404d3b35f1"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.576438 10912 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 20:28:29.577653 10927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:29.577905 10912 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:29.578037 10912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "16d0e63367c04226b2d004c1807ae126"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "790e44aabad30dd61e0614a9e9df20a3"
server_key_iv: "372420e1b4dc9cce16329f404d3b35f1"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.578142 10912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:29.612160 10912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:29.614441 10912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:29.614645 10912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:29.619477 10912 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:36861
I20260928 20:28:29.619531 10980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:36861 every 8 connection(s)
I20260928 20:28:29.619992 10912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:29.620893 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:29.623232 10981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: Bootstrap starting.
I20260928 20:28:29.623914 10981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:29.624255 10981 log.cc:824] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:29.625092 10981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: No bootstrap required, opened a new log
I20260928 20:28:29.626966 10981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } }
I20260928 20:28:29.627137 10981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:29.627162 10981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d0e63367c04226b2d004c1807ae126, State: Initialized, Role: FOLLOWER
I20260928 20:28:29.627362 10981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } }
I20260928 20:28:29.627421 10981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:29.627444 10981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:29.627499 10981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:29.628168 10981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } }
I20260928 20:28:29.628285 10981 leader_election.cc:302] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d0e63367c04226b2d004c1807ae126; no voters:
I20260928 20:28:29.628525 10981 leader_election.cc:288] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:29.628572 10984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:29.628806 10981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:29.628850 10984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [term 1 LEADER]: Becoming Leader. State: Replica: 16d0e63367c04226b2d004c1807ae126, State: Running, Role: LEADER
I20260928 20:28:29.628948 10984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } }
I20260928 20:28:29.629550 10986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d0e63367c04226b2d004c1807ae126. Latest consensus state: current_term: 1 leader_uuid: "16d0e63367c04226b2d004c1807ae126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } } }
I20260928 20:28:29.629643 10986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:29.629690 10985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d0e63367c04226b2d004c1807ae126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d0e63367c04226b2d004c1807ae126" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36861 } } }
I20260928 20:28:29.629783 10985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:29.629887 10993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:29.630028 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 10912
I20260928 20:28:29.630170 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:29.630389 25323 external_mini_cluster.cc:1428] Setting key 53246e8090f927fc342c3e83c3f50a89
I20260928 20:28:29.630570 10993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:29.632536 10993 catalog_manager.cc:1347] Generated new cluster ID: cdd11f4211404dd98289012062d9fc13
I20260928 20:28:29.632596 10993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:29.641474 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:29.631023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.24.186.254:36861)
0928 20:28:29.631499 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:29.631508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:29.631539 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:28:29.631671 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:29.631678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:29.631736 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:29.631946 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:29.632826 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.633900 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.634752 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.635225 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.635353 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:29.637804 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:29.637810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:29.637810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:29.637826 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:29.639715 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:29.640126 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:29.640130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:29.640130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:29.640245 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:29.640465 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:29.640466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:29.640467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:29.640748 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:29.640924 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:29.641025 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 20:28:29.641286 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":66,"threads_started":1}
I20260928 20:28:29.648578 10993 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:29.649345 10993 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:29.655920 10993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: Generated new TSK 0
I20260928 20:28:29.656180 10993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:28:29.690980 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:29.776662 11007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:29.776880 11007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:29.776906 11007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:29.778028 11007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:29.778065 11007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:29.778091 11007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:29.779486 11007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:29.779696 11007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:29.779965 11007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:29.785539 11013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.785588 11015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.785544 11012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:29.786635 11007 server_base.cc:1034] running on GCE node
I20260928 20:28:29.786904 11007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:29.787209 11007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:29.788651 11007 hybrid_clock.cc:648] HybridClock initialized: now 1790627309788638 us; error 30 us; skew 500 ppm
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:29.792310 11007 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:28:29.793134 11007 webserver.cc:460] Webserver started at http://127.24.186.193:38631/ using document root <none> and password file <none>
I20260928 20:28:29.793392 11007 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:29.793443 11007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:29.793555 11007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:29.794673 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "f3d1fdc4b38f408e9d982190972485eb"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "93b9813354a3ad21851835eb1c7cd31c"
server_key_iv: "e710ba6616aabb6d7d0ee0d845867b90"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.794987 11007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "f3d1fdc4b38f408e9d982190972485eb"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "93b9813354a3ad21851835eb1c7cd31c"
server_key_iv: "e710ba6616aabb6d7d0ee0d845867b90"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.796522 11007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:29.797422 11022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:29.797804 11007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:29.797981 11007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "f3d1fdc4b38f408e9d982190972485eb"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "93b9813354a3ad21851835eb1c7cd31c"
server_key_iv: "e710ba6616aabb6d7d0ee0d845867b90"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.798053 11007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:29.831972 11007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:29.834354 11007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:29.834484 11007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:29.834717 11007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:29.835049 11007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:29.835089 11007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:29.835114 11007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:29.835127 11007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:29.841647 11007 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:46743
I20260928 20:28:29.841809 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:46743 every 8 connection(s)
I20260928 20:28:29.842083 11007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:29.847189 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11007
I20260928 20:28:29.847368 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:29.847579 25323 external_mini_cluster.cc:1428] Setting key b993ab197e89870baf321fc13656f936
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:29.854833 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:29.842972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:46753 (local address 127.24.186.254:36861)
0928 20:28:29.843050 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:29.843052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:29.843218 (+ 166us) server_negotiation.cc:407] Connection header received
0928 20:28:29.843710 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:29.843711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:29.843725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:29.843790 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:29.844828 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.845404 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.846555 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.846773 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.846848 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:29.850668 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:29.850669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:29.850670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:29.850677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:29.852462 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:29.853244 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:29.853246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:29.853247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:29.853340 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:29.853915 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:29.853916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:29.853917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:29.854085 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:29.854186 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:29.854622 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 20:28:29.854732 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:28:29.855085 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:29.842920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36861 (local address 127.24.186.193:46753)
0928 20:28:29.843086 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:29.843131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:29.843609 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:29.843913 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:29.843917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:29.844497 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:29.844701 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:29.844706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.845503 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.845504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:29.846447 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:29.846450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:29.846895 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:29.846896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:29.847120 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:29.848111 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:29.848123 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:29.850522 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:29.852625 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:29.852640 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:29.852644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:29.853076 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:29.853626 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:29.853628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:29.853629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:29.853820 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:29.854174 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:29.854182 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:29.854442 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 20:28:29.854596 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 20:28:29.854802 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 20:28:29.855616 11137 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36861
I20260928 20:28:29.855732 11137 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:29.855947 11137 heartbeater.cc:511] Master 127.24.186.254:36861 requested a full tablet report, sending...
I20260928 20:28:29.856693 10944 ts_manager.cc:194] Registered new tserver with Master: f3d1fdc4b38f408e9d982190972485eb (127.24.186.193:46743)
I20260928 20:28:29.857292 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:46753
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:28:29.878504 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:29.966143 11142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:29.966375 11142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:29.966404 11142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:29.967649 11142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:29.967696 11142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:29.967724 11142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:29.969308 11142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:29.969578 11142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:29.969847 11142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:29.975756 11147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.975792 11148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:29.975756 11150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:29.976003 11142 server_base.cc:1034] running on GCE node
I20260928 20:28:29.976367 11142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:29.976673 11142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:29.977864 11142 hybrid_clock.cc:648] HybridClock initialized: now 1790627309977850 us; error 47 us; skew 500 ppm
Sep 28 20:28:29 dist-test-slave-k477 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:29.980721 11142 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:28:29.981341 11142 webserver.cc:460] Webserver started at http://127.24.186.194:38295/ using document root <none> and password file <none>
I20260928 20:28:29.981652 11142 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:29.981730 11142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:29.981901 11142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:29.983209 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "f77bc57c0cdb422da9615db11e71d673"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "fdc3c5f56a168ad291117dbec1e0c504"
server_key_iv: "c1d975200e5d2c97e22a43bca0865fdf"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.983985 11142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "f77bc57c0cdb422da9615db11e71d673"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "fdc3c5f56a168ad291117dbec1e0c504"
server_key_iv: "c1d975200e5d2c97e22a43bca0865fdf"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.985577 11142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:29.986720 11157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:29.987032 11142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:29.987141 11142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "f77bc57c0cdb422da9615db11e71d673"
format_stamp: "Formatted at 2026-09-28 20:28:29 on dist-test-slave-k477"
server_key: "fdc3c5f56a168ad291117dbec1e0c504"
server_key_iv: "c1d975200e5d2c97e22a43bca0865fdf"
server_key_version: "encryptionkey@0"
I20260928 20:28:29.987236 11142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:29.998867 11142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:30.001425 11142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:30.001613 11142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:30.001871 11142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:30.002213 11142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:30.002272 11142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.002315 11142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:30.002342 11142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.009704 11142 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:44883
I20260928 20:28:30.009752 11271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:44883 every 8 connection(s)
I20260928 20:28:30.010082 11142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:30.015971 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11142
I20260928 20:28:30.016129 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:30.016500 25323 external_mini_cluster.cc:1428] Setting key d7e9efdf403ca0f8bb3b5794ebcaef2e
Sep 28 20:28:30 dist-test-slave-k477 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627309, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:30.022423 11274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.010812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36861 (local address 127.24.186.194:35015)
0928 20:28:30.011014 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.011089 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.011626 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.011999 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.012003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.012637 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:30.012850 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.012854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.013571 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.013572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.014969 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.014973 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.015524 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.015525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.015694 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.016633 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:30.016650 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:30.018594 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:30.020424 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:30.020432 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:30.020435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:30.020776 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:30.021011 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:30.021011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:30.021012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:30.021187 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:30.021514 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:30.021521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:30.021729 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.021870 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.022089 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260928 20:28:30.022825 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.010878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:35015 (local address 127.24.186.254:36861)
0928 20:28:30.010931 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.010938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.011167 (+ 229us) server_negotiation.cc:407] Connection header received
0928 20:28:30.011732 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.011733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.011743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.011889 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.012954 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.013467 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.015086 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.015413 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.015483 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.018710 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.018712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.018713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:30.018721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.020326 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.020864 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.020865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.020865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.020935 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.021293 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.021294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.021294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.021426 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.021502 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.022298 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.022706 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:28:30.023550 11272 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36861
I20260928 20:28:30.023651 11272 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:30.023947 11272 heartbeater.cc:511] Master 127.24.186.254:36861 requested a full tablet report, sending...
I20260928 20:28:30.024468 10944 ts_manager.cc:194] Registered new tserver with Master: f77bc57c0cdb422da9615db11e71d673 (127.24.186.194:44883)
I20260928 20:28:30.025004 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:35015
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:28:30.044641 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:30.133433 11277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:30.133685 11277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:30.133713 11277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:30.134894 11277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:30.134933 11277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:30.134961 11277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:30.136545 11277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:30.136780 11277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:30.136983 11277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:30.142704 11285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.142683 11283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.142675 11282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.143584 11277 server_base.cc:1034] running on GCE node
I20260928 20:28:30.144233 11277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:30.144558 11277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:30.145758 11277 hybrid_clock.cc:648] HybridClock initialized: now 1790627310145738 us; error 44 us; skew 500 ppm
Sep 28 20:28:30 dist-test-slave-k477 krb5kdc[10899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627310, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:30.148872 11277 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:28:30.149405 11277 webserver.cc:460] Webserver started at http://127.24.186.195:44335/ using document root <none> and password file <none>
I20260928 20:28:30.149662 11277 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:30.149744 11277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:30.149900 11277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:30.151414 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "840045d604d44dfeb5336d61173997a7"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8d6725de2220dda489f54001ec8f674d"
server_key_iv: "51ba683dd6883ff2b2437d99aa7d3243"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.151991 11277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "840045d604d44dfeb5336d61173997a7"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8d6725de2220dda489f54001ec8f674d"
server_key_iv: "51ba683dd6883ff2b2437d99aa7d3243"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.153707 11277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:28:30.154853 11292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.155117 11277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:30.155232 11277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "840045d604d44dfeb5336d61173997a7"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8d6725de2220dda489f54001ec8f674d"
server_key_iv: "51ba683dd6883ff2b2437d99aa7d3243"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.155334 11277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:30.169411 11277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:30.173226 11277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:30.173412 11277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:30.173791 11277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:30.174234 11277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:30.174297 11277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.174347 11277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:30.174376 11277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.182132 11277 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:35399
I20260928 20:28:30.182238 11406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:35399 every 8 connection(s)
I20260928 20:28:30.182602 11277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
Sep 28 20:28:30 dist-test-slave-k477 krb5kdc[10899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627310, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:30.191006 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11277
I20260928 20:28:30.191160 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:30.191596 25323 external_mini_cluster.cc:1428] Setting key a74d0ff4080af78ea3df6a2bc6a54d67
I20260928 20:28:30.195763 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.183543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:44293 (local address 127.24.186.254:36861)
0928 20:28:30.183681 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.183684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.184079 (+ 395us) server_negotiation.cc:407] Connection header received
0928 20:28:30.184690 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.184692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.184704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.184754 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.186026 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.186545 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.187788 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.188046 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.188138 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.191046 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.191050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.191050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:30.191066 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.193421 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.194248 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.194250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.194251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.194332 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.194667 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.194669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.194669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.194843 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.194935 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.195510 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.195654 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 20:28:30.196018 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.183397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36861 (local address 127.24.186.195:44293)
0928 20:28:30.183681 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.183767 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.184572 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.184840 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.184844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.185625 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:30.185887 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.185893 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.186662 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.186663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.187633 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.187639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.188150 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.188152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.188308 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.188878 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:30.188888 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:30.190845 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:30.193553 (+ 2708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:30.193572 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:30.193582 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:30.194111 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:30.194421 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:30.194422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:30.194422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:30.194580 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:30.194922 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:30.194930 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:30.195243 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.195458 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.195664 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260928 20:28:30.196719 11407 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36861
I20260928 20:28:30.196828 11407 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:30.196996 11407 heartbeater.cc:511] Master 127.24.186.254:36861 requested a full tablet report, sending...
I20260928 20:28:30.197352 10944 ts_manager.cc:194] Registered new tserver with Master: 840045d604d44dfeb5336d61173997a7 (127.24.186.195:35399)
I20260928 20:28:30.197772 10944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:44293
I20260928 20:28:30.205041 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:30.215008 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.207515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.24.186.254:36861)
0928 20:28:30.207607 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.207610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.207654 (+ 44us) server_negotiation.cc:407] Connection header received
0928 20:28:30.207685 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.207686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.207704 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.207774 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.208414 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.208918 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.209668 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.209861 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.209938 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.211912 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.211914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.211915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:30.211927 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.213638 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.214127 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.214129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.214129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.214190 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:30.214423 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:30.214424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:30.214424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:30.214572 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.214692 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.214835 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.214899 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:28:30.217056 10944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:30.218793 10944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:30.226692 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:35399 (local address 127.0.0.1:56152)
0928 20:28:30.222192 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.222238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.222333 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.222686 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.222691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.222694 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:30.222769 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.222771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.224153 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.224154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.225412 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.225415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226385 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.226457 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226462 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.226493 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.226569 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":16,"thread_start_us":88,"threads_started":1}
I20260928 20:28:30.226747 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.24.186.193:46743)
0928 20:28:30.222660 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.222665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.222684 (+ 19us) server_negotiation.cc:407] Connection header received
0928 20:28:30.222721 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.222805 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.222825 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.222901 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:30.223092 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.224170 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.225724 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226404 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226471 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226526 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.226602 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.226648 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260928 20:28:30.226747 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.24.186.195:35399)
0928 20:28:30.222491 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.222495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.222506 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:28:30.222572 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.222574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.222590 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.222671 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:30.222871 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.224039 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.225518 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226273 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226541 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226597 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.226621 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.226660 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":103,"threads_started":1}
I20260928 20:28:30.226938 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:46743 (local address 127.0.0.1:35610)
0928 20:28:30.222378 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.222388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.222432 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.222974 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.222975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.222977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:30.223019 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.223021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.224401 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.224403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.225606 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.225609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226472 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.226530 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226533 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.226577 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.226606 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260928 20:28:30.227228 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:44883 (local address 127.0.0.1:53798)
0928 20:28:30.222786 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.222802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.222853 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.223067 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.223068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.223070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:30.223117 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.223119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.224053 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.224054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.225602 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.225606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226314 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.226393 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226408 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.226478 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.227157 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260928 20:28:30.226791 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.222378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.24.186.194:44883)
0928 20:28:30.222561 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.222566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.222913 (+ 347us) server_negotiation.cc:407] Connection header received
0928 20:28:30.222956 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.222959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.222973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.223069 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:30.223178 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.223952 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.225719 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.226217 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.226318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.226378 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.226648 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.226707 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260928 20:28:30.227706 11341 tablet_service.cc:1467] Processing CreateTablet for tablet e211b895fedd4f25be54dff553861bf5 (DEFAULT_TABLE table=test-table [id=c86c758214084160ab9472954664eab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:30.227742 11071 tablet_service.cc:1467] Processing CreateTablet for tablet e211b895fedd4f25be54dff553861bf5 (DEFAULT_TABLE table=test-table [id=c86c758214084160ab9472954664eab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:30.227929 11206 tablet_service.cc:1467] Processing CreateTablet for tablet e211b895fedd4f25be54dff553861bf5 (DEFAULT_TABLE table=test-table [id=c86c758214084160ab9472954664eab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:30.228066 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211b895fedd4f25be54dff553861bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:30.228108 11071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211b895fedd4f25be54dff553861bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:30.228204 11206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e211b895fedd4f25be54dff553861bf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:30.231537 11424 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Bootstrap starting.
I20260928 20:28:30.232105 11426 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Bootstrap starting.
I20260928 20:28:30.232342 11424 tablet_bootstrap.cc:654] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:30.232724 11426 tablet_bootstrap.cc:654] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:30.232884 11424 log.cc:824] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:30.233320 11426 log.cc:824] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:30.233702 11425 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Bootstrap starting.
I20260928 20:28:30.233829 11424 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: No bootstrap required, opened a new log
I20260928 20:28:30.233918 11424 ts_tablet_manager.cc:1397] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:30.234129 11426 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: No bootstrap required, opened a new log
I20260928 20:28:30.234196 11426 ts_tablet_manager.cc:1397] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:30.234361 11425 tablet_bootstrap.cc:654] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:30.234910 11425 log.cc:824] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:30.235610 11425 tablet_bootstrap.cc:492] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: No bootstrap required, opened a new log
I20260928 20:28:30.235707 11425 ts_tablet_manager.cc:1397] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 20:28:30.236102 11424 raft_consensus.cc:348] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.236192 11426 raft_consensus.cc:348] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.236310 11424 raft_consensus.cc:374] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:30.236342 11426 raft_consensus.cc:374] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:30.236351 11424 raft_consensus.cc:729] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840045d604d44dfeb5336d61173997a7, State: Initialized, Role: FOLLOWER
I20260928 20:28:30.236385 11426 raft_consensus.cc:729] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f77bc57c0cdb422da9615db11e71d673, State: Initialized, Role: FOLLOWER
I20260928 20:28:30.236490 11426 consensus_queue.cc:260] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.236490 11424 consensus_queue.cc:260] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.236789 11424 ts_tablet_manager.cc:1428] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:30.236789 11426 ts_tablet_manager.cc:1428] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:30.236909 11407 heartbeater.cc:503] Master 127.24.186.254:36861 was elected leader, sending a full tablet report...
I20260928 20:28:30.236953 11272 heartbeater.cc:503] Master 127.24.186.254:36861 was elected leader, sending a full tablet report...
I20260928 20:28:30.237713 11425 raft_consensus.cc:348] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.237888 11425 raft_consensus.cc:374] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:30.237933 11425 raft_consensus.cc:729] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d1fdc4b38f408e9d982190972485eb, State: Initialized, Role: FOLLOWER
I20260928 20:28:30.238057 11425 consensus_queue.cc:260] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.238406 11425 ts_tablet_manager.cc:1428] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:28:30.238407 11137 heartbeater.cc:503] Master 127.24.186.254:36861 was elected leader, sending a full tablet report...
W20260928 20:28:30.260855 11273 tablet.cc:2369] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:30.319216 11431 raft_consensus.cc:482] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:30.319442 11431 raft_consensus.cc:504] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.319934 11431 leader_election.cc:288] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3d1fdc4b38f408e9d982190972485eb (127.24.186.193:46743), f77bc57c0cdb422da9615db11e71d673 (127.24.186.194:44883)
I20260928 20:28:30.323793 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.320301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:44449 (local address 127.24.186.193:46743)
0928 20:28:30.320347 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.320349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.320379 (+ 30us) server_negotiation.cc:407] Connection header received
0928 20:28:30.320411 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.320413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.320422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.320485 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:30.320757 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.321524 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.322996 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.323383 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.323462 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.323474 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.323670 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.323718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:28:30.323793 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.320144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:46743 (local address 127.24.186.195:44449)
0928 20:28:30.320235 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.320261 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.320335 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.320565 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.320567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.320570 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:30.320648 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.320651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.321619 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.321620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.322879 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.322882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.323485 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.323486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.323562 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.323566 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.323663 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.323711 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 20:28:30.324195 11091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211b895fedd4f25be54dff553861bf5" candidate_uuid: "840045d604d44dfeb5336d61173997a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d1fdc4b38f408e9d982190972485eb" is_pre_election: true
I20260928 20:28:30.324402 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.320254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:44883 (local address 127.24.186.195:56919)
0928 20:28:30.320439 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.320448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.320500 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.320759 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.320760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.320763 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:30.320836 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.320837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.321779 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.321780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.322879 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.322882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.324151 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.324153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.324224 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.324228 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.324293 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.324336 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
I20260928 20:28:30.324394 11091 raft_consensus.cc:2454] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840045d604d44dfeb5336d61173997a7 in term 0.
I20260928 20:28:30.324465 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.320339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:56919 (local address 127.24.186.194:44883)
0928 20:28:30.320383 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.320386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.320542 (+ 156us) server_negotiation.cc:407] Connection header received
0928 20:28:30.320572 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.320573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.320584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.320661 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:30.321003 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.321698 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.323510 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.324064 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.324137 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.324144 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.324335 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.324382 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:28:30.324627 11296 leader_election.cc:302] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 840045d604d44dfeb5336d61173997a7, f3d1fdc4b38f408e9d982190972485eb; no voters:
I20260928 20:28:30.324759 11431 raft_consensus.cc:2781] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:30.324759 11226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211b895fedd4f25be54dff553861bf5" candidate_uuid: "840045d604d44dfeb5336d61173997a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77bc57c0cdb422da9615db11e71d673" is_pre_election: true
I20260928 20:28:30.324838 11431 raft_consensus.cc:482] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:30.324865 11431 raft_consensus.cc:3037] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:30.324901 11226 raft_consensus.cc:2454] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 840045d604d44dfeb5336d61173997a7 in term 0.
I20260928 20:28:30.325686 11431 raft_consensus.cc:504] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.325834 11431 leader_election.cc:288] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [CANDIDATE]: Term 1 election: Requested vote from peers f3d1fdc4b38f408e9d982190972485eb (127.24.186.193:46743), f77bc57c0cdb422da9615db11e71d673 (127.24.186.194:44883)
I20260928 20:28:30.325996 11091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211b895fedd4f25be54dff553861bf5" candidate_uuid: "840045d604d44dfeb5336d61173997a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3d1fdc4b38f408e9d982190972485eb"
I20260928 20:28:30.326076 11091 raft_consensus.cc:3037] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:30.326066 11226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e211b895fedd4f25be54dff553861bf5" candidate_uuid: "840045d604d44dfeb5336d61173997a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f77bc57c0cdb422da9615db11e71d673"
I20260928 20:28:30.326167 11226 raft_consensus.cc:3037] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:30.327244 11091 raft_consensus.cc:2454] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840045d604d44dfeb5336d61173997a7 in term 1.
I20260928 20:28:30.327294 11226 raft_consensus.cc:2454] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 840045d604d44dfeb5336d61173997a7 in term 1.
I20260928 20:28:30.327665 11296 leader_election.cc:302] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 840045d604d44dfeb5336d61173997a7, f77bc57c0cdb422da9615db11e71d673; no voters:
I20260928 20:28:30.327855 11431 raft_consensus.cc:2781] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:30.328102 11431 raft_consensus.cc:686] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 1 LEADER]: Becoming Leader. State: Replica: 840045d604d44dfeb5336d61173997a7, State: Running, Role: LEADER
I20260928 20:28:30.328346 11431 consensus_queue.cc:237] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } }
I20260928 20:28:30.329666 10943 catalog_manager.cc:5495] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 840045d604d44dfeb5336d61173997a7 (127.24.186.195). New cstate: current_term: 1 leader_uuid: "840045d604d44dfeb5336d61173997a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "840045d604d44dfeb5336d61173997a7" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 35399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 } health_report { overall_health: UNKNOWN } } }
W20260928 20:28:30.343043 11138 tablet.cc:2369] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:30.351637 11420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.348657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.24.186.195:35399)
0928 20:28:30.348763 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.348766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.348779 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:28:30.348871 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.348872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.348902 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.348988 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:30.349151 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.349631 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.350404 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.350757 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.350894 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.351156 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:30.351284 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:30.351410 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.351488 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.351503 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 20:28:30.354028 11091 raft_consensus.cc:1261] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 1 FOLLOWER]: Refusing update from remote peer 840045d604d44dfeb5336d61173997a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:30.354097 11226 raft_consensus.cc:1261] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 1 FOLLOWER]: Refusing update from remote peer 840045d604d44dfeb5336d61173997a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:30.354651 11434 consensus_queue.cc:1035] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3d1fdc4b38f408e9d982190972485eb" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:30.354897 11431 consensus_queue.cc:1035] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f77bc57c0cdb422da9615db11e71d673" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:30.358654 11439 mvcc.cc:205] Tried to move back new op lower bound from 7334409463206805504 to 7334409463105961984. Current Snapshot: MvccSnapshot[applied={T|T < 7334409463206805504 or (T in {7334409463206805504})}]
I20260928 20:28:30.360534 10943 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:30.360632 10943 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:30.361786 10943 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16d0e63367c04226b2d004c1807ae126: Sending DeleteTablet for 3 replicas of tablet e211b895fedd4f25be54dff553861bf5
I20260928 20:28:30.362113 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet e211b895fedd4f25be54dff553861bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:30 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:35610
I20260928 20:28:30.362154 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet e211b895fedd4f25be54dff553861bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:30 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56152
I20260928 20:28:30.362182 11206 tablet_service.cc:1514] Processing DeleteTablet for tablet e211b895fedd4f25be54dff553861bf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:30 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:53798
I20260928 20:28:30.362520 11448 tablet_replica.cc:330] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: stopping tablet replica
I20260928 20:28:30.362562 11449 tablet_replica.cc:330] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: stopping tablet replica
I20260928 20:28:30.362624 11448 raft_consensus.cc:2229] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:28:30.362643 11449 raft_consensus.cc:2229] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:30.362723 11448 raft_consensus.cc:2258] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:30.362741 11449 raft_consensus.cc:2258] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:30.362872 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11007
I20260928 20:28:30.362937 11447 tablet_replica.cc:330] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb: stopping tablet replica
I20260928 20:28:30.363042 11447 raft_consensus.cc:2229] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:30.363142 11447 raft_consensus.cc:2258] T e211b895fedd4f25be54dff553861bf5 P f3d1fdc4b38f408e9d982190972485eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:30.364785 11449 ts_tablet_manager.cc:1905] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:30.366909 11449 ts_tablet_manager.cc:1918] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:30.366935 11448 ts_tablet_manager.cc:1905] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:30.367095 11449 log.cc:1196] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/e211b895fedd4f25be54dff553861bf5
I20260928 20:28:30.367489 11449 ts_tablet_manager.cc:1939] T e211b895fedd4f25be54dff553861bf5 P f77bc57c0cdb422da9615db11e71d673: Deleting consensus metadata
I20260928 20:28:30.368065 10931 catalog_manager.cc:4841] TS f77bc57c0cdb422da9615db11e71d673 (127.24.186.194:44883): tablet e211b895fedd4f25be54dff553861bf5 (table test-table [id=c86c758214084160ab9472954664eab6]) successfully deleted
I20260928 20:28:30.369195 11448 ts_tablet_manager.cc:1918] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:30.369350 11448 log.cc:1196] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/e211b895fedd4f25be54dff553861bf5
I20260928 20:28:30.369896 11448 ts_tablet_manager.cc:1939] T e211b895fedd4f25be54dff553861bf5 P 840045d604d44dfeb5336d61173997a7: Deleting consensus metadata
I20260928 20:28:30.370496 10928 catalog_manager.cc:4841] TS 840045d604d44dfeb5336d61173997a7 (127.24.186.195:35399): tablet e211b895fedd4f25be54dff553861bf5 (table test-table [id=c86c758214084160ab9472954664eab6]) successfully deleted
W20260928 20:28:30.371240 10931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:46743 (error 108)
I20260928 20:28:30.371568 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11142
I20260928 20:28:30.377948 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11277
I20260928 20:28:30.378150 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.377924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:46743 (local address 127.0.0.1:35616)
0928 20:28:30.378010 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.378079 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 20:28:30.379567 10931 catalog_manager.cc:4584] TS f3d1fdc4b38f408e9d982190972485eb (127.24.186.193:46743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e211b895fedd4f25be54dff553861bf5: Network error: Client connection negotiation failed: client connection to 127.24.186.193:46743: connect: Connection refused (error 111)
I20260928 20:28:30.383378 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 10912
2026-09-28T20:28:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2126 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T20:28:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:30Z Disabled control of system clock
I20260928 20:28:30.418295 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:39537
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:30.513468 11454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:30.513734 11454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:30.513757 11454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:30.515018 11454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:30.515076 11454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:30.515089 11454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:30.515102 11454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:30.515115 11454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:30.516674 11454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:39537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:30.516966 11454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:30.517313 11454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:30.522926 11460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.523030 11454 server_base.cc:1034] running on GCE node
W20260928 20:28:30.523123 11462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.522948 11459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.523433 11454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:30.523749 11454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:30.524966 11454 hybrid_clock.cc:648] HybridClock initialized: now 1790627310524955 us; error 31 us; skew 500 ppm
I20260928 20:28:30.532742 11454 webserver.cc:460] Webserver started at http://127.24.186.254:38201/ using document root <none> and password file <none>
I20260928 20:28:30.533080 11454 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:30.533166 11454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:30.533326 11454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:30.534482 11454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "1219830ed6c64769a3e8a883518dafe4"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "c88e33338e15d3945f87ea94777b5c14"
server_key_iv: "4ec3e6c8897318408acd444cdfd1c63d"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.534883 11454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "1219830ed6c64769a3e8a883518dafe4"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "c88e33338e15d3945f87ea94777b5c14"
server_key_iv: "4ec3e6c8897318408acd444cdfd1c63d"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.536736 11454 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 20:28:30.537796 11469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.537994 11454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:30.538107 11454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "1219830ed6c64769a3e8a883518dafe4"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "c88e33338e15d3945f87ea94777b5c14"
server_key_iv: "4ec3e6c8897318408acd444cdfd1c63d"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.538192 11454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:30.557003 11454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:30.557382 11454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:30.557551 11454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:30.561667 11454 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:39537
I20260928 20:28:30.561707 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:39537 every 8 connection(s)
I20260928 20:28:30.562077 11454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:30.563089 11523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:30.564632 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11454
I20260928 20:28:30.564796 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:30.565150 25323 external_mini_cluster.cc:1428] Setting key e2a41919a43ff9be75adc0be5d51763e
I20260928 20:28:30.566402 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: Bootstrap starting.
I20260928 20:28:30.567154 11523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:30.567529 11523 log.cc:824] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:30.568403 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: No bootstrap required, opened a new log
I20260928 20:28:30.570739 11523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } }
I20260928 20:28:30.570952 11523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:30.570990 11523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1219830ed6c64769a3e8a883518dafe4, State: Initialized, Role: FOLLOWER
I20260928 20:28:30.571152 11523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } }
I20260928 20:28:30.571285 11523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:30.571332 11523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:30.571389 11523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:30.572075 11523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } }
I20260928 20:28:30.572209 11523 leader_election.cc:302] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1219830ed6c64769a3e8a883518dafe4; no voters:
I20260928 20:28:30.572456 11523 leader_election.cc:288] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:30.572485 11528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:30.572608 11528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [term 1 LEADER]: Becoming Leader. State: Replica: 1219830ed6c64769a3e8a883518dafe4, State: Running, Role: LEADER
I20260928 20:28:30.572705 11528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } }
I20260928 20:28:30.572785 11523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:30.573150 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1219830ed6c64769a3e8a883518dafe4. Latest consensus state: current_term: 1 leader_uuid: "1219830ed6c64769a3e8a883518dafe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } } }
I20260928 20:28:30.573477 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:30.573522 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.566066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.24.186.254:39537)
0928 20:28:30.566337 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.566346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.566379 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:28:30.566991 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.567001 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.567084 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.567259 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.567537 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.568904 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.569724 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.570459 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.570596 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.571441 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.571445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.571446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:30.572325 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.572426 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:30.572453 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.573059 (+ 606us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.573114 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.573248 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":125,"threads_started":1}
I20260928 20:28:30.573407 11530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1219830ed6c64769a3e8a883518dafe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1219830ed6c64769a3e8a883518dafe4" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 39537 } } }
I20260928 20:28:30.573643 11530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:30.574046 11537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:30.574728 11537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:30.576735 11537 catalog_manager.cc:1347] Generated new cluster ID: 2dd6aaa99bb240d9b5eb129c224f8a81
I20260928 20:28:30.576799 11537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:30.592967 11537 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:30.593716 11537 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:30.599987 11537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: Generated new TSK 0
I20260928 20:28:30.600242 11537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:30.606065 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--builtin_ntp_servers=127.24.186.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:30.695904 11547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:30.696146 11547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:30.696171 11547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:30.697508 11547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:30.697563 11547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:30.697592 11547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:30.699070 11547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:30.699354 11547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:30.699690 11547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:30.705256 11552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.705468 11555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.705266 11553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.706087 11547 server_base.cc:1034] running on GCE node
I20260928 20:28:30.706413 11547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:30.706732 11547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:30.707901 11547 hybrid_clock.cc:648] HybridClock initialized: now 1790627310707876 us; error 41 us; skew 500 ppm
I20260928 20:28:30.710450 11547 webserver.cc:460] Webserver started at http://127.24.186.193:40917/ using document root <none> and password file <none>
I20260928 20:28:30.710767 11547 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:30.710824 11547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:30.710970 11547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:30.712752 11547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "d3b38c5d0c234f79adedc893f3898fb2"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8b2420c1a00220c1d80038cf4fa85c91"
server_key_iv: "9d6e8a57a9d300300a5113a973a1f331"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.713376 11547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "d3b38c5d0c234f79adedc893f3898fb2"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8b2420c1a00220c1d80038cf4fa85c91"
server_key_iv: "9d6e8a57a9d300300a5113a973a1f331"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.715114 11547 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:28:30.715992 11561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.716148 11547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:30.716261 11547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "d3b38c5d0c234f79adedc893f3898fb2"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "8b2420c1a00220c1d80038cf4fa85c91"
server_key_iv: "9d6e8a57a9d300300a5113a973a1f331"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.716346 11547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:30.746137 11547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:30.746501 11547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:30.746662 11547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:30.746932 11547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:30.747295 11547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:30.747355 11547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.747399 11547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:30.747428 11547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.753865 11547 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:36871
I20260928 20:28:30.753926 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:36871 every 8 connection(s)
I20260928 20:28:30.754303 11547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:30.759697 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.755107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:55981 (local address 127.24.186.254:39537)
0928 20:28:30.755176 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.755178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.755439 (+ 261us) server_negotiation.cc:407] Connection header received
0928 20:28:30.755967 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.755968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.755979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.756048 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.756726 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.757248 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.758259 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.758441 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.758495 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.759387 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.759389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.759389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:30.759394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.759409 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:30.759413 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.759520 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.759607 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.759630 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 20:28:30.759961 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.755083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:39537 (local address 127.24.186.193:55981)
0928 20:28:30.755298 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.755341 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.755863 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.756109 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.756112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.756120 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:30.756637 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.756641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.757355 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.757356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.758164 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.758167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.758537 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.758538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.758663 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.759205 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:30.759205 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:30.759235 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:30.759266 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:30.759523 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:30.759528 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:30.759530 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.759591 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.759682 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260928 20:28:30.760722 11676 heartbeater.cc:347] Connected to a master server at 127.24.186.254:39537
I20260928 20:28:30.760842 11676 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:30.761053 11676 heartbeater.cc:511] Master 127.24.186.254:39537 requested a full tablet report, sending...
I20260928 20:28:30.761749 11487 ts_manager.cc:194] Registered new tserver with Master: d3b38c5d0c234f79adedc893f3898fb2 (127.24.186.193:36871)
I20260928 20:28:30.762396 11487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.193:55981
I20260928 20:28:30.762745 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11547
I20260928 20:28:30.762843 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:30.763106 25323 external_mini_cluster.cc:1428] Setting key a10e0aeb8a280aebf22a12e5658276bb
I20260928 20:28:30.764726 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--builtin_ntp_servers=127.24.186.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:30.853813 11679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:30.854051 11679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:30.854076 11679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:30.855221 11679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:30.855259 11679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:30.855285 11679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:30.856637 11679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:30.856864 11679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:30.857131 11679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:30.863312 11684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:30.863529 11685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.863605 11679 server_base.cc:1034] running on GCE node
W20260928 20:28:30.863529 11687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:30.864056 11679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:30.864368 11679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:30.865576 11679 hybrid_clock.cc:648] HybridClock initialized: now 1790627310865549 us; error 41 us; skew 500 ppm
I20260928 20:28:30.867055 11679 webserver.cc:460] Webserver started at http://127.24.186.194:36871/ using document root <none> and password file <none>
I20260928 20:28:30.867364 11679 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:30.867440 11679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:30.867597 11679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:30.869318 11679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "19238d19eb32fa5555ad5b156ec6ebcb"
server_key_iv: "30628d4d6993ae3491d9d0918e0fbb15"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.869748 11679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "19238d19eb32fa5555ad5b156ec6ebcb"
server_key_iv: "30628d4d6993ae3491d9d0918e0fbb15"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.871208 11679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:30.872117 11693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.872362 11679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:30.872452 11679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395"
format_stamp: "Formatted at 2026-09-28 20:28:30 on dist-test-slave-k477"
server_key: "19238d19eb32fa5555ad5b156ec6ebcb"
server_key_iv: "30628d4d6993ae3491d9d0918e0fbb15"
server_key_version: "encryptionkey@0"
I20260928 20:28:30.872534 11679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:30.882411 11679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:30.882915 11679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:30.883248 11679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:30.883601 11679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:30.884040 11679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:30.884078 11679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.884100 11679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:30.884146 11679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:30.891286 11679 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:46873
I20260928 20:28:30.891341 11807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:46873 every 8 connection(s)
I20260928 20:28:30.891682 11679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:30.897641 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.892507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:49339 (local address 127.24.186.254:39537)
0928 20:28:30.892556 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:30.892559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:30.892842 (+ 283us) server_negotiation.cc:407] Connection header received
0928 20:28:30.893403 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:30.893404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:30.893415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:30.893482 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:30.894218 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.894746 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.895729 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.896012 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.896090 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.897199 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:30.897201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:30.897202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:30.897210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:30.897236 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:30.897242 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:30.897337 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:30.897537 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 20:28:30.897571 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:28:30.897820 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:30.892508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:39537 (local address 127.24.186.194:49339)
0928 20:28:30.892723 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:30.892757 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:30.893303 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:30.893549 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:30.893552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:30.893560 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:30.894143 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.894148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.894826 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.894827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.895633 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:30.895636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:30.896129 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:30.896131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:30.896347 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:30.896954 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:30.896955 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:30.896991 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:30.897017 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:30.897393 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:30.897399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:30.897401 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:28:30.897484 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 20:28:30.897581 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260928 20:28:30.898447 11808 heartbeater.cc:347] Connected to a master server at 127.24.186.254:39537
I20260928 20:28:30.898545 11808 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:30.898799 11808 heartbeater.cc:511] Master 127.24.186.254:39537 requested a full tablet report, sending...
I20260928 20:28:30.899276 11487 ts_manager.cc:194] Registered new tserver with Master: 442cf3faaa6c4a0fbf5cae69cd8e3395 (127.24.186.194:46873)
I20260928 20:28:30.899873 11487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.194:49339
I20260928 20:28:30.901507 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11679
I20260928 20:28:30.901651 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:30.901863 25323 external_mini_cluster.cc:1428] Setting key 3309a733c118d07f7f87713f44ecc1e1
I20260928 20:28:30.902817 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--builtin_ntp_servers=127.24.186.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:30.994388 11811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:30.994692 11811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:30.994737 11811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:30.996098 11811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:30.996168 11811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:30.996214 11811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:30.997692 11811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:39537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:30.997996 11811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:30.998498 11811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:31.003753 11819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:31.003753 11817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:31.003793 11816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:31.003914 11811 server_base.cc:1034] running on GCE node
I20260928 20:28:31.004307 11811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:31.004561 11811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:31.005738 11811 hybrid_clock.cc:648] HybridClock initialized: now 1790627311005707 us; error 48 us; skew 500 ppm
I20260928 20:28:31.007483 11811 webserver.cc:460] Webserver started at http://127.24.186.195:37609/ using document root <none> and password file <none>
I20260928 20:28:31.007766 11811 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:31.007856 11811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:31.007979 11811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:31.009164 11811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "fdbd3bb309ec43e29f567857fbbf8b3b"
format_stamp: "Formatted at 2026-09-28 20:28:31 on dist-test-slave-k477"
server_key: "9eac2e47345cbd7c48642f790e253cd3"
server_key_iv: "cba6d0b3cfdd07507777d7896d452715"
server_key_version: "encryptionkey@0"
I20260928 20:28:31.009554 11811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "fdbd3bb309ec43e29f567857fbbf8b3b"
format_stamp: "Formatted at 2026-09-28 20:28:31 on dist-test-slave-k477"
server_key: "9eac2e47345cbd7c48642f790e253cd3"
server_key_iv: "cba6d0b3cfdd07507777d7896d452715"
server_key_version: "encryptionkey@0"
I20260928 20:28:31.011030 11811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:31.011991 11825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:31.012181 11811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:31.012270 11811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "fdbd3bb309ec43e29f567857fbbf8b3b"
format_stamp: "Formatted at 2026-09-28 20:28:31 on dist-test-slave-k477"
server_key: "9eac2e47345cbd7c48642f790e253cd3"
server_key_iv: "cba6d0b3cfdd07507777d7896d452715"
server_key_version: "encryptionkey@0"
I20260928 20:28:31.012342 11811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:31.032346 11811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:31.032663 11811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:31.032790 11811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:31.032989 11811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:31.033459 11811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:31.033514 11811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:31.033540 11811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:31.033587 11811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:31.039710 11811 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45011
I20260928 20:28:31.039794 11939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45011 every 8 connection(s)
I20260928 20:28:31.040169 11811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:28:31.046792 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.041087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:40771 (local address 127.24.186.254:39537)
0928 20:28:31.041135 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.041137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.041438 (+ 301us) server_negotiation.cc:407] Connection header received
0928 20:28:31.042269 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.042274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.042285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.042344 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:31.043087 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.043727 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.045175 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.045386 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.045449 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.046371 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:31.046372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:31.046372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:31.046378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:31.046394 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:31.046397 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:31.046451 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.046696 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.046723 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:28:31.047250 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.041112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:39537 (local address 127.24.186.195:40771)
0928 20:28:31.041292 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.041328 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.042138 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.042407 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.042410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.042419 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:31.043008 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.043020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.043896 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.043897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.045082 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.045085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.045491 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.045492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.045616 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.046202 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:31.046203 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:31.046235 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:31.046260 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:31.046540 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:31.046549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:31.046554 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.046677 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.046816 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260928 20:28:31.048005 11940 heartbeater.cc:347] Connected to a master server at 127.24.186.254:39537
I20260928 20:28:31.048133 11940 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:31.048338 11940 heartbeater.cc:511] Master 127.24.186.254:39537 requested a full tablet report, sending...
I20260928 20:28:31.048743 11487 ts_manager.cc:194] Registered new tserver with Master: fdbd3bb309ec43e29f567857fbbf8b3b (127.24.186.195:45011)
I20260928 20:28:31.049207 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11811
I20260928 20:28:31.049378 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:31.049527 11487 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.195:40771
I20260928 20:28:31.049810 25323 external_mini_cluster.cc:1428] Setting key b486046d1e769756624e0553240f16f9
I20260928 20:28:31.050765 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:31.055738 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.052238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.24.186.254:39537)
0928 20:28:31.052304 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.052306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.052418 (+ 112us) server_negotiation.cc:407] Connection header received
0928 20:28:31.052466 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.052467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.052484 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.052574 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:31.052843 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.053365 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.054133 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.054349 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.054412 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.055191 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:31.055193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:31.055193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:31.055199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:31.055225 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:31.055229 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:31.055359 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.055628 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.055648 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 20:28:31.058166 11487 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:31.060078 11487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:31.067344 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.063280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.24.186.195:45011)
0928 20:28:31.063437 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.063441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.063846 (+ 405us) server_negotiation.cc:407] Connection header received
0928 20:28:31.063919 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.063921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.063937 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.064009 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.064207 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065281 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.066366 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.066881 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.066987 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.067083 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.067206 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.067249 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260928 20:28:31.067332 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.063002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45011 (local address 127.0.0.1:45808)
0928 20:28:31.063731 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.063740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.063786 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.064051 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.064052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.064054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.064108 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.064110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065377 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.065378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.066277 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.066280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.066987 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.066988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.067066 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.067079 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.067190 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.067247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 20:28:31.067631 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.062935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36871 (local address 127.0.0.1:60224)
0928 20:28:31.063470 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.063511 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.063599 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.063824 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.063827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.063829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.063922 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.063925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065107 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.065108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.066380 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.066382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.067391 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.067392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.067454 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.067459 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.067509 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.067556 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":83,"threads_started":1}
I20260928 20:28:31.067931 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.063185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.24.186.193:36871)
0928 20:28:31.063375 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.063382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.063589 (+ 207us) server_negotiation.cc:407] Connection header received
0928 20:28:31.063689 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.063691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.063709 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.063903 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.063997 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065012 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.066473 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.067198 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.067262 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.067745 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.067773 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.067851 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260928 20:28:31.068285 11873 tablet_service.cc:1467] Processing CreateTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 (DEFAULT_TABLE table=test-table [id=8b81fd8002d448c8ada6360e5fe735a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:31.068511 11609 tablet_service.cc:1467] Processing CreateTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 (DEFAULT_TABLE table=test-table [id=8b81fd8002d448c8ada6360e5fe735a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:31.068766 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.063011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46873 (local address 127.0.0.1:60632)
0928 20:28:31.064096 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.064105 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.064162 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.064452 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.064454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.064460 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.064594 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.064601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065856 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.065858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.066897 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.066899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.068222 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.068224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.068379 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.068389 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.068628 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.068696 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":75,"threads_started":1}
I20260928 20:28:31.068810 11609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d51e4e0af554295a6d8f81109bc8980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:31.068802 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d51e4e0af554295a6d8f81109bc8980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:31.068941 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.063239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.24.186.194:46873)
0928 20:28:31.063441 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.063446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.064205 (+ 759us) server_negotiation.cc:407] Connection header received
0928 20:28:31.064253 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.064255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.064271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.064359 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.064708 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.065757 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.066988 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.068069 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.068149 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.068233 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.068685 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.068750 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260928 20:28:31.069476 11741 tablet_service.cc:1467] Processing CreateTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 (DEFAULT_TABLE table=test-table [id=8b81fd8002d448c8ada6360e5fe735a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:31.069767 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d51e4e0af554295a6d8f81109bc8980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:31.072476 11955 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Bootstrap starting.
I20260928 20:28:31.072795 11954 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Bootstrap starting.
I20260928 20:28:31.073196 11955 tablet_bootstrap.cc:654] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:31.073529 11955 log.cc:824] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:31.073683 11954 tablet_bootstrap.cc:654] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:31.073906 11956 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: Bootstrap starting.
I20260928 20:28:31.074033 11954 log.cc:824] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:31.074553 11956 tablet_bootstrap.cc:654] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:31.074829 11956 log.cc:824] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:31.075194 11954 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: No bootstrap required, opened a new log
I20260928 20:28:31.075263 11955 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: No bootstrap required, opened a new log
I20260928 20:28:31.075291 11954 ts_tablet_manager.cc:1397] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:28:31.075333 11955 ts_tablet_manager.cc:1397] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 20:28:31.077375 11955 raft_consensus.cc:348] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.077545 11955 raft_consensus.cc:374] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:31.077570 11955 raft_consensus.cc:729] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbd3bb309ec43e29f567857fbbf8b3b, State: Initialized, Role: FOLLOWER
I20260928 20:28:31.077492 11954 raft_consensus.cc:348] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.077696 11954 raft_consensus.cc:374] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:31.077706 11955 consensus_queue.cc:260] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.077746 11954 raft_consensus.cc:729] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442cf3faaa6c4a0fbf5cae69cd8e3395, State: Initialized, Role: FOLLOWER
I20260928 20:28:31.077857 11954 consensus_queue.cc:260] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.078033 11940 heartbeater.cc:503] Master 127.24.186.254:39537 was elected leader, sending a full tablet report...
I20260928 20:28:31.078154 11808 heartbeater.cc:503] Master 127.24.186.254:39537 was elected leader, sending a full tablet report...
I20260928 20:28:31.078464 11954 ts_tablet_manager.cc:1428] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:28:31.078019 11955 ts_tablet_manager.cc:1428] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:31.079731 11956 tablet_bootstrap.cc:492] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: No bootstrap required, opened a new log
I20260928 20:28:31.079838 11956 ts_tablet_manager.cc:1397] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 20:28:31.082209 11956 raft_consensus.cc:348] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.082427 11956 raft_consensus.cc:374] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:31.082464 11956 raft_consensus.cc:729] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b38c5d0c234f79adedc893f3898fb2, State: Initialized, Role: FOLLOWER
I20260928 20:28:31.082588 11956 consensus_queue.cc:260] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.082877 11956 ts_tablet_manager.cc:1428] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:31.082952 11676 heartbeater.cc:503] Master 127.24.186.254:39537 was elected leader, sending a full tablet report...
I20260928 20:28:31.107144 11962 raft_consensus.cc:482] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:31.107385 11962 raft_consensus.cc:504] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.107788 11962 leader_election.cc:288] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdbd3bb309ec43e29f567857fbbf8b3b (127.24.186.195:45011), 442cf3faaa6c4a0fbf5cae69cd8e3395 (127.24.186.194:46873)
I20260928 20:28:31.109851 11960 raft_consensus.cc:482] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:31.109978 11960 raft_consensus.cc:504] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.110358 11960 leader_election.cc:288] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3b38c5d0c234f79adedc893f3898fb2 (127.24.186.193:36871), fdbd3bb309ec43e29f567857fbbf8b3b (127.24.186.195:45011)
I20260928 20:28:31.111960 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.108125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:35323 (local address 127.24.186.194:46873)
0928 20:28:31.108186 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.108188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.108197 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:31.108236 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.108236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.108246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.108345 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.108567 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.109397 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111064 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111603 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111663 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.111671 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.111853 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.111895 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:28:31.111971 11953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.108127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:54475 (local address 127.24.186.195:45011)
0928 20:28:31.108178 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.108181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.108311 (+ 130us) server_negotiation.cc:407] Connection header received
0928 20:28:31.108372 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.108374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.108384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.108444 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.108612 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.109396 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.110931 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111509 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111572 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.111580 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.111875 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.111914 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:28:31.112231 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.108022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:46873 (local address 127.24.186.193:35323)
0928 20:28:31.108079 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.108093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.108159 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.108366 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.108367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.108368 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.108465 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.108467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.109520 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.109521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.110963 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.110965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111696 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.111758 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.111761 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.112082 (+ 321us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.112139 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":215}
I20260928 20:28:31.112239 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.108034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45011 (local address 127.24.186.193:54475)
0928 20:28:31.108232 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.108241 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.108294 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.108460 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.108460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.108461 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.108519 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.108520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.109520 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.109521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.110824 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.110828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111676 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.111748 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.111752 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.112081 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.112138 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":86,"thread_start_us":107,"threads_started":1}
I20260928 20:28:31.112557 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "d3b38c5d0c234f79adedc893f3898fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" is_pre_election: true
I20260928 20:28:31.112576 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "d3b38c5d0c234f79adedc893f3898fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" is_pre_election: true
I20260928 20:28:31.112679 11761 raft_consensus.cc:2454] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b38c5d0c234f79adedc893f3898fb2 in term 0.
I20260928 20:28:31.112706 11893 raft_consensus.cc:2454] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3b38c5d0c234f79adedc893f3898fb2 in term 0.
I20260928 20:28:31.112893 11562 leader_election.cc:302] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b38c5d0c234f79adedc893f3898fb2, fdbd3bb309ec43e29f567857fbbf8b3b; no voters:
I20260928 20:28:31.113018 11962 raft_consensus.cc:2781] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:31.113072 11962 raft_consensus.cc:482] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:31.113089 11962 raft_consensus.cc:3037] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:31.114099 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.110503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36871 (local address 127.24.186.194:58441)
0928 20:28:31.110604 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.110615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.110668 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.110931 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.110932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.110934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.111009 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.111011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111922 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.111922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.113092 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.113095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.113826 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.113827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.113906 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.113910 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.113971 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.114019 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 20:28:31.114037 11962 raft_consensus.cc:504] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.114142 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.110750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:58441 (local address 127.24.186.193:36871)
0928 20:28:31.110792 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.110795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.110803 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:28:31.110825 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.110826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.110837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.110888 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.111087 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.111837 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.113185 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.113709 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.113782 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.113790 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.113991 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.114031 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:28:31.114200 11962 leader_election.cc:288] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [CANDIDATE]: Term 1 election: Requested vote from peers fdbd3bb309ec43e29f567857fbbf8b3b (127.24.186.195:45011), 442cf3faaa6c4a0fbf5cae69cd8e3395 (127.24.186.194:46873)
I20260928 20:28:31.114456 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "d3b38c5d0c234f79adedc893f3898fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b"
I20260928 20:28:31.114571 11893 raft_consensus.cc:3037] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:31.114782 11964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.110529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45011 (local address 127.24.186.194:34157)
0928 20:28:31.110822 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.110831 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:31.110874 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:31.111118 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:31.111118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:31.111120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:31.111191 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.111192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.112581 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.112582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.113598 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:31.113601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.114527 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.114528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:31.114597 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.114601 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:31.114660 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 20:28:31.114704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20260928 20:28:31.114795 11965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.110638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:34157 (local address 127.24.186.195:45011)
0928 20:28:31.110986 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.110988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.110996 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:28:31.111016 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.111017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.111027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.111254 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:31.111286 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.112491 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.113735 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.114455 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.114523 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.114530 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.114655 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.114724 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":76,"threads_started":1}
I20260928 20:28:31.115020 11630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b38c5d0c234f79adedc893f3898fb2" is_pre_election: true
I20260928 20:28:31.115090 11761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "d3b38c5d0c234f79adedc893f3898fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395"
I20260928 20:28:31.115154 11630 raft_consensus.cc:2379] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 442cf3faaa6c4a0fbf5cae69cd8e3395 in current term 1: Already voted for candidate d3b38c5d0c234f79adedc893f3898fb2 in this term.
I20260928 20:28:31.115157 11761 raft_consensus.cc:3037] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:31.115207 11892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d51e4e0af554295a6d8f81109bc8980" candidate_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" is_pre_election: true
I20260928 20:28:31.115521 11696 leader_election.cc:302] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 442cf3faaa6c4a0fbf5cae69cd8e3395; no voters: d3b38c5d0c234f79adedc893f3898fb2, fdbd3bb309ec43e29f567857fbbf8b3b
I20260928 20:28:31.116076 11761 raft_consensus.cc:2454] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b38c5d0c234f79adedc893f3898fb2 in term 1.
I20260928 20:28:31.116077 11893 raft_consensus.cc:2454] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3b38c5d0c234f79adedc893f3898fb2 in term 1.
I20260928 20:28:31.116179 11960 raft_consensus.cc:2726] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 20:28:31.116232 11562 leader_election.cc:302] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3b38c5d0c234f79adedc893f3898fb2, fdbd3bb309ec43e29f567857fbbf8b3b; no voters:
I20260928 20:28:31.116314 11962 raft_consensus.cc:2781] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:31.116472 11962 raft_consensus.cc:686] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 LEADER]: Becoming Leader. State: Replica: d3b38c5d0c234f79adedc893f3898fb2, State: Running, Role: LEADER
I20260928 20:28:31.116580 11962 consensus_queue.cc:237] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } }
I20260928 20:28:31.117503 11486 catalog_manager.cc:5495] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3b38c5d0c234f79adedc893f3898fb2 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "d3b38c5d0c234f79adedc893f3898fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b38c5d0c234f79adedc893f3898fb2" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 } health_report { overall_health: UNKNOWN } } }
I20260928 20:28:31.130573 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.128039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.24.186.193:36871)
0928 20:28:31.128097 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:31.128099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:31.128108 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:31.128208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:31.128209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:31.128236 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:31.128316 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:31.128438 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.128926 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.129689 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:31.129884 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:31.129960 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:31.130111 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:31.130210 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:31.130417 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:31.130473 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 20:28:31.130488 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:28:31.132671 11893 raft_consensus.cc:1261] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 1 FOLLOWER]: Refusing update from remote peer d3b38c5d0c234f79adedc893f3898fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:31.132679 11761 raft_consensus.cc:1261] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 1 FOLLOWER]: Refusing update from remote peer d3b38c5d0c234f79adedc893f3898fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:31.133033 11962 consensus_queue.cc:1035] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "442cf3faaa6c4a0fbf5cae69cd8e3395" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:31.133199 11966 consensus_queue.cc:1035] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdbd3bb309ec43e29f567857fbbf8b3b" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:31.136063 11970 mvcc.cc:205] Tried to move back new op lower bound from 7334409466396647424 to 7334409466334224384. Current Snapshot: MvccSnapshot[applied={T|T < 7334409466396647424}]
I20260928 20:28:31.138195 11486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:31.138306 11486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:31.139051 11486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1219830ed6c64769a3e8a883518dafe4: Sending DeleteTablet for 3 replicas of tablet 2d51e4e0af554295a6d8f81109bc8980
I20260928 20:28:31.139405 11609 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:31 UTC) from {username='slave'} at 127.0.0.1:60224
I20260928 20:28:31.139431 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:31 UTC) from {username='slave'} at 127.0.0.1:60632
I20260928 20:28:31.139724 11979 tablet_replica.cc:330] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2: stopping tablet replica
I20260928 20:28:31.139854 11979 raft_consensus.cc:2229] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:28:31.139856 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d51e4e0af554295a6d8f81109bc8980 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:31 UTC) from {username='slave'} at 127.0.0.1:45808
I20260928 20:28:31.139994 11979 raft_consensus.cc:2258] T 2d51e4e0af554295a6d8f81109bc8980 P d3b38c5d0c234f79adedc893f3898fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:31.140141 11981 tablet_replica.cc:330] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: stopping tablet replica
I20260928 20:28:31.140204 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11547
I20260928 20:28:31.140230 11980 tablet_replica.cc:330] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: stopping tablet replica
I20260928 20:28:31.140424 11981 raft_consensus.cc:2229] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:31.140618 11981 raft_consensus.cc:2258] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:31.141916 11981 ts_tablet_manager.cc:1905] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:31.142130 11980 raft_consensus.cc:2229] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:31.142261 11980 raft_consensus.cc:2258] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:31.142656 11980 ts_tablet_manager.cc:1905] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:31.144133 11980 ts_tablet_manager.cc:1918] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:31.144233 11980 log.cc:1196] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/2d51e4e0af554295a6d8f81109bc8980
I20260928 20:28:31.144357 11981 ts_tablet_manager.cc:1918] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:31.144444 11981 log.cc:1196] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/2d51e4e0af554295a6d8f81109bc8980
I20260928 20:28:31.144527 11980 ts_tablet_manager.cc:1939] T 2d51e4e0af554295a6d8f81109bc8980 P 442cf3faaa6c4a0fbf5cae69cd8e3395: Deleting consensus metadata
I20260928 20:28:31.144719 11981 ts_tablet_manager.cc:1939] T 2d51e4e0af554295a6d8f81109bc8980 P fdbd3bb309ec43e29f567857fbbf8b3b: Deleting consensus metadata
I20260928 20:28:31.145063 11473 catalog_manager.cc:4841] TS 442cf3faaa6c4a0fbf5cae69cd8e3395 (127.24.186.194:46873): tablet 2d51e4e0af554295a6d8f81109bc8980 (table test-table [id=8b81fd8002d448c8ada6360e5fe735a9]) successfully deleted
I20260928 20:28:31.145125 11470 catalog_manager.cc:4841] TS fdbd3bb309ec43e29f567857fbbf8b3b (127.24.186.195:45011): tablet 2d51e4e0af554295a6d8f81109bc8980 (table test-table [id=8b81fd8002d448c8ada6360e5fe735a9]) successfully deleted
W20260928 20:28:31.146570 11472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:36871 (error 108)
I20260928 20:28:31.147100 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:31.146971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36871 (local address 127.0.0.1:60246)
0928 20:28:31.147031 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:31.147070 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:36871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:28:31.147226 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11679
W20260928 20:28:31.147269 11472 catalog_manager.cc:4584] TS d3b38c5d0c234f79adedc893f3898fb2 (127.24.186.193:36871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d51e4e0af554295a6d8f81109bc8980: Network error: Client connection negotiation failed: client connection to 127.24.186.193:36871: connect: Connection refused (error 111)
I20260928 20:28:31.152737 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11811
I20260928 20:28:31.157586 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11454
2026-09-28T20:28:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (770 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:31 dist-test-slave-k477 krb5kdc[11985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:31 dist-test-slave-k477 krb5kdc[11985](info): set up 2 sockets
Sep 28 20:28:31 dist-test-slave-k477 krb5kdc[11985](info): commencing operation
krb5kdc: starting...
W20260928 20:28:32.217074 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:32.344916 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:43257
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:43257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:32.433153 11998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:32.433666 11998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:32.433706 11998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:32.435312 11998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:32.435395 11998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:32.435410 11998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:32.435424 11998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:32.435439 11998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:32.437037 11998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:43257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:32.437254 11998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:32.437469 11998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:32.442564 12006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:32.442564 12004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:32.442600 12003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:32.442744 11998 server_base.cc:1034] running on GCE node
I20260928 20:28:32.443006 11998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:32.443334 11998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:32.444476 11998 hybrid_clock.cc:648] HybridClock initialized: now 1790627312444462 us; error 33 us; skew 500 ppm
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:32.447081 11998 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:32.447491 11998 webserver.cc:460] Webserver started at http://127.24.186.254:37709/ using document root <none> and password file <none>
I20260928 20:28:32.447692 11998 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:32.447732 11998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:32.447885 11998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:32.448917 11998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "cb228541119842879bab829c1d1e83b1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ee0e2640997400f27fd5e152920b8057"
server_key_iv: "225ddce52c46e4a90fa1bd82d7a5d967"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.449219 11998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "cb228541119842879bab829c1d1e83b1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ee0e2640997400f27fd5e152920b8057"
server_key_iv: "225ddce52c46e4a90fa1bd82d7a5d967"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.450559 11998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:32.451259 12013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.451416 11998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:28:32.451505 11998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "cb228541119842879bab829c1d1e83b1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ee0e2640997400f27fd5e152920b8057"
server_key_iv: "225ddce52c46e4a90fa1bd82d7a5d967"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.451576 11998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:32.475521 11998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:32.478040 11998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:32.478161 11998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:32.482105 11998 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:43257
I20260928 20:28:32.482195 12065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:43257 every 8 connection(s)
I20260928 20:28:32.482553 11998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:32.483824 12066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:32.486202 12066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: Bootstrap starting.
I20260928 20:28:32.486809 12066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:32.487102 12066 log.cc:824] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:32.487872 12066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: No bootstrap required, opened a new log
I20260928 20:28:32.489574 12066 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } }
I20260928 20:28:32.489748 12066 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:32.489843 12066 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb228541119842879bab829c1d1e83b1, State: Initialized, Role: FOLLOWER
I20260928 20:28:32.490048 12066 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } }
I20260928 20:28:32.490164 12066 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:32.490209 12066 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:32.490263 12066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:32.490854 12066 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } }
I20260928 20:28:32.490983 12066 leader_election.cc:302] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb228541119842879bab829c1d1e83b1; no voters:
I20260928 20:28:32.491196 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 11998
I20260928 20:28:32.491230 12066 leader_election.cc:288] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:32.491321 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:32.491361 12069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:32.491513 25323 external_mini_cluster.cc:1428] Setting key c4240c6ab35e2ad855ffcb78b821aa7d
I20260928 20:28:32.491508 12066 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:32.491560 12069 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [term 1 LEADER]: Becoming Leader. State: Replica: cb228541119842879bab829c1d1e83b1, State: Running, Role: LEADER
I20260928 20:28:32.491675 12069 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } }
I20260928 20:28:32.492216 12070 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb228541119842879bab829c1d1e83b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } } }
I20260928 20:28:32.492390 12070 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:32.492624 12071 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb228541119842879bab829c1d1e83b1. Latest consensus state: current_term: 1 leader_uuid: "cb228541119842879bab829c1d1e83b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb228541119842879bab829c1d1e83b1" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 43257 } } }
I20260928 20:28:32.492697 12071 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:32.492894 12080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:32.493566 12080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:32.495292 12080 catalog_manager.cc:1347] Generated new cluster ID: c0c58d6fb7424feeba1731391d9ed682
I20260928 20:28:32.495349 12080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:32.501186 12080 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:32.501735 12080 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:32.505550 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:32.492017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.24.186.254:43257)
0928 20:28:32.492894 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:32.492904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:32.493101 (+ 197us) server_negotiation.cc:407] Connection header received
0928 20:28:32.493399 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:32.493407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:32.493497 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:32.494715 (+ 1218us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:32.495112 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.496569 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.497962 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.498654 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.498786 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:32.501782 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:32.501786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:32.501786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:32.501800 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:32.503852 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.504281 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.504284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.504285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.504389 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.504608 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.504609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.504609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.504885 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:32.505035 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:32.505166 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:28:32.505316 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":91,"threads_started":1}
I20260928 20:28:32.515623 12080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: Generated new TSK 0
I20260928 20:28:32.516165 12080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.24.186.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
Entry for principal kudu/127.24.186.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab.
I20260928 20:28:32.555967 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:32.650406 12092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:32.650728 12092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:32.650763 12092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:32.652917 12092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:32.652995 12092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:32.653036 12092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:32.654712 12092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.193.keytab
--principal=kudu/127.24.186.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:32.654986 12092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:32.655254 12092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:32.660662 12098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:32.660866 12092 server_base.cc:1034] running on GCE node
W20260928 20:28:32.660727 12097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:32.660655 12100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:32.661193 12092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:32.661453 12092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:32.662631 12092 hybrid_clock.cc:648] HybridClock initialized: now 1790627312662604 us; error 37 us; skew 500 ppm
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:32.665678 12092 init.cc:377] Logged in from keytab as kudu/127.24.186.193@KRBTEST.COM (short username kudu)
I20260928 20:28:32.666162 12092 webserver.cc:460] Webserver started at http://127.24.186.193:46337/ using document root <none> and password file <none>
I20260928 20:28:32.666400 12092 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:32.666497 12092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:32.666651 12092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:32.667724 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "943997bf31914ebd88175e16ebd6a5f1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "373638498af4be07a4617663c653bc15"
server_key_iv: "ee2a075bfd825087189b035d139d083c"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.668147 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "943997bf31914ebd88175e16ebd6a5f1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "373638498af4be07a4617663c653bc15"
server_key_iv: "ee2a075bfd825087189b035d139d083c"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.669503 12092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:32.670331 12107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.670533 12092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:32.670634 12092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "943997bf31914ebd88175e16ebd6a5f1"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "373638498af4be07a4617663c653bc15"
server_key_iv: "ee2a075bfd825087189b035d139d083c"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.670727 12092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:32.690285 12092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:32.692658 12092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:32.692829 12092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:32.693560 12092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:32.693961 12092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:32.694021 12092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.694056 12092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:32.694087 12092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.700636 12092 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:39895
I20260928 20:28:32.700712 12220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:39895 every 8 connection(s)
I20260928 20:28:32.701061 12092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:32.702198 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12092
I20260928 20:28:32.702327 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:32.702544 25323 external_mini_cluster.cc:1428] Setting key 1d1c1263a0de942d8e4b5c49ec79963f
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.193@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.194@KRBTEST.COM; defaulting to no policy
I20260928 20:28:32.711920 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:32.701909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:49917 (local address 127.24.186.254:43257)
0928 20:28:32.701985 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:32.701988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:32.702293 (+ 305us) server_negotiation.cc:407] Connection header received
0928 20:28:32.702988 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:32.702989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:32.703002 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:32.703044 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:32.704454 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.704987 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.706043 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.706256 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.706341 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:32.708755 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:32.708757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:32.708757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:32.708765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:32.710298 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.710767 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.710768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.710769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.710856 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.711178 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.711179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.711179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.711294 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:32.711348 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:32.711663 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 20:28:32.711781 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 20:28:32.712132 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:32.701900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:43257 (local address 127.24.186.193:49917)
0928 20:28:32.702114 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:32.702170 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:32.702885 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:32.703120 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:32.703125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:32.704031 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:32.704330 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:32.704337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.705125 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.705126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:32.705929 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:32.705931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.706351 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.706352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:32.706494 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:32.707131 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:32.707140 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:32.708621 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:32.710421 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:32.710430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:32.710432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:32.710684 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:32.710940 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:32.710941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:32.710942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:32.711102 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:32.711377 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:32.711388 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:32.711524 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 20:28:32.711681 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 20:28:32.711903 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260928 20:28:32.712766 12221 heartbeater.cc:347] Connected to a master server at 127.24.186.254:43257
I20260928 20:28:32.712853 12221 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:32.713002 12221 heartbeater.cc:511] Master 127.24.186.254:43257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.194@KRBTEST.COM" created.
I20260928 20:28:32.713502 12030 ts_manager.cc:194] Registered new tserver with Master: 943997bf31914ebd88175e16ebd6a5f1 (127.24.186.193:39895)
I20260928 20:28:32.714432 12030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.193@KRBTEST.COM'} at 127.24.186.193:49917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
Entry for principal kudu/127.24.186.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab.
I20260928 20:28:32.730844 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:32.827718 12226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:32.828060 12226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:32.828090 12226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:32.829332 12226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:32.829386 12226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:32.829416 12226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:32.830946 12226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.194.keytab
--principal=kudu/127.24.186.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:32.831205 12226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:32.831450 12226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:32.837054 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:32.837095 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:32.837342 12226 server_base.cc:1034] running on GCE node
W20260928 20:28:32.837061 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:32.837596 12226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:32.837873 12226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:32.839031 12226 hybrid_clock.cc:648] HybridClock initialized: now 1790627312839021 us; error 34 us; skew 500 ppm
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:32.841589 12226 init.cc:377] Logged in from keytab as kudu/127.24.186.194@KRBTEST.COM (short username kudu)
I20260928 20:28:32.842073 12226 webserver.cc:460] Webserver started at http://127.24.186.194:38725/ using document root <none> and password file <none>
I20260928 20:28:32.842314 12226 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:32.842382 12226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:32.842537 12226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:32.843863 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "5c899c2d05864382b8ef0eef00c53913"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ac78aa4e024b3af6b0e8a7d8143f8659"
server_key_iv: "afea3572fe831fdf17622c66a6fca8ff"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.844448 12226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "5c899c2d05864382b8ef0eef00c53913"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ac78aa4e024b3af6b0e8a7d8143f8659"
server_key_iv: "afea3572fe831fdf17622c66a6fca8ff"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.845973 12226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 20:28:32.846776 12241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.846976 12226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:32.847095 12226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "5c899c2d05864382b8ef0eef00c53913"
format_stamp: "Formatted at 2026-09-28 20:28:32 on dist-test-slave-k477"
server_key: "ac78aa4e024b3af6b0e8a7d8143f8659"
server_key_iv: "afea3572fe831fdf17622c66a6fca8ff"
server_key_version: "encryptionkey@0"
I20260928 20:28:32.847167 12226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:32.868305 12226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:32.871203 12226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:32.871336 12226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:32.871675 12226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:32.872094 12226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:32.872157 12226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.872193 12226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:32.872222 12226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:32.878965 12226 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:41993
I20260928 20:28:32.879027 12354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:41993 every 8 connection(s)
I20260928 20:28:32.879372 12226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
Sep 28 20:28:32 dist-test-slave-k477 krb5kdc[11985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627312, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.194@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:32.887194 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12226
I20260928 20:28:32.887535 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:32.887778 25323 external_mini_cluster.cc:1428] Setting key 86528064286110dc9ac28df23e15ac73
I20260928 20:28:32.890266 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:32.880265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:47069 (local address 127.24.186.254:43257)
0928 20:28:32.880351 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:32.880354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:32.880555 (+ 201us) server_negotiation.cc:407] Connection header received
0928 20:28:32.881029 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:32.881030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:32.881040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:32.881113 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:32.882140 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.882699 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.883934 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.884177 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.884248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:32.886995 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:32.886997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:32.886997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:32.887005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:32.888745 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.889214 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.889215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.889215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.889281 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:32.889550 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:32.889550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:32.889551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:32.889673 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:32.889745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:32.890127 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 20:28:32.890200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:28:32.890578 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:32.880205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:43257 (local address 127.24.186.194:47069)
0928 20:28:32.880420 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:32.880461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:32.880935 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:32.881172 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:32.881175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:32.881756 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:32.882027 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:32.882032 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.882803 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.882804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:32.883856 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:32.883859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:32.884305 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:32.884306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:32.884429 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:32.884964 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:32.884973 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:32.886795 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:32.888839 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:32.888850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:32.888853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:32.889130 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:32.889361 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:32.889361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:32.889362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:32.889475 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:32.889746 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:32.889753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:32.889961 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 20:28:32.890095 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 20:28:32.890293 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
I20260928 20:28:32.891153 12355 heartbeater.cc:347] Connected to a master server at 127.24.186.254:43257
I20260928 20:28:32.891302 12355 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:32.891506 12355 heartbeater.cc:511] Master 127.24.186.254:43257 requested a full tablet report, sending...
I20260928 20:28:32.891916 12030 ts_manager.cc:194] Registered new tserver with Master: 5c899c2d05864382b8ef0eef00c53913 (127.24.186.194:41993)
I20260928 20:28:32.892401 12030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.194@KRBTEST.COM'} at 127.24.186.194:47069
WARNING: no policy specified for kudu/127.24.186.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
Entry for principal kudu/127.24.186.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab.
I20260928 20:28:32.915110 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:37479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:32.999297 12360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:32.999555 12360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:32.999653 12360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:33.000926 12360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:33.000996 12360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:33.001044 12360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:33.002534 12360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:37479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.195.keytab
--principal=kudu/127.24.186.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:33.002810 12360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:33.003046 12360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:33.008281 12368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.008299 12365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.008329 12360 server_base.cc:1034] running on GCE node
W20260928 20:28:33.008299 12366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.008662 12360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:33.008929 12360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:33.010159 12360 hybrid_clock.cc:648] HybridClock initialized: now 1790627313010140 us; error 26 us; skew 500 ppm
Sep 28 20:28:33 dist-test-slave-k477 krb5kdc[11985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627313, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:33.013249 12360 init.cc:377] Logged in from keytab as kudu/127.24.186.195@KRBTEST.COM (short username kudu)
I20260928 20:28:33.013789 12360 webserver.cc:460] Webserver started at http://127.24.186.195:45191/ using document root <none> and password file <none>
I20260928 20:28:33.014022 12360 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:33.014101 12360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:33.014254 12360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:33.015458 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "0288d4b514134ceb958cc40ccb17a9ba"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "e7a021ae5286f624176ca017a3765540"
server_key_iv: "6837cc887134688c8841cde89a9783bc"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.015905 12360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "0288d4b514134ceb958cc40ccb17a9ba"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "e7a021ae5286f624176ca017a3765540"
server_key_iv: "6837cc887134688c8841cde89a9783bc"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.017349 12360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:33.018100 12375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.018272 12360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.018376 12360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "0288d4b514134ceb958cc40ccb17a9ba"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "e7a021ae5286f624176ca017a3765540"
server_key_iv: "6837cc887134688c8841cde89a9783bc"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.018451 12360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:33.044888 12360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:33.047160 12360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:33.047351 12360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:33.047621 12360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:33.048043 12360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:33.048097 12360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.048148 12360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:33.048177 12360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.054612 12360 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:34269
I20260928 20:28:33.054723 12489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:34269 every 8 connection(s)
I20260928 20:28:33.055055 12360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:28:33.060760 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12360
I20260928 20:28:33.060844 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:33.061040 25323 external_mini_cluster.cc:1428] Setting key cd8a0b8478acdc0e3d468a3d895c7f6a
Sep 28 20:28:33 dist-test-slave-k477 krb5kdc[11985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627313, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.195@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:33.065640 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.056309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:40093 (local address 127.24.186.254:43257)
0928 20:28:33.056388 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.056390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.056551 (+ 161us) server_negotiation.cc:407] Connection header received
0928 20:28:33.057070 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.057072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.057085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.057146 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.058092 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.058603 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.059585 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.059795 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.059910 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.062597 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.062598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.062599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:33.062607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.064168 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:33.064630 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:33.064631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:33.064631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:33.064694 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:33.064966 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:33.064967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:33.064967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:33.065088 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.065162 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.065448 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.065545 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 20:28:33.065851 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.056264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:43257 (local address 127.24.186.195:40093)
0928 20:28:33.056424 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.056466 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.056958 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.057214 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.057218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.057797 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:33.057989 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.057993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.058702 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.058703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.059489 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.059492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.059920 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.059920 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.060050 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.060561 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:28:33.060569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:33.062491 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:33.064274 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:33.064284 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:33.064286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:33.064541 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:33.064768 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:28:33.064769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:28:33.064769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:28:33.064891 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:28:33.065165 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:33.065172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:33.065316 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.065426 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.065599 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260928 20:28:33.066524 12490 heartbeater.cc:347] Connected to a master server at 127.24.186.254:43257
I20260928 20:28:33.066675 12490 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:33.066895 12490 heartbeater.cc:511] Master 127.24.186.254:43257 requested a full tablet report, sending...
I20260928 20:28:33.067338 12030 ts_manager.cc:194] Registered new tserver with Master: 0288d4b514134ceb958cc40ccb17a9ba (127.24.186.195:34269)
I20260928 20:28:33.067757 12030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.24.186.195@KRBTEST.COM'} at 127.24.186.195:40093
I20260928 20:28:33.072794 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:33.081714 12074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.074212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44628 (local address 127.24.186.254:43257)
0928 20:28:33.074281 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.074284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.074425 (+ 141us) server_negotiation.cc:407] Connection header received
0928 20:28:33.074481 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.074481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.074491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.074534 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.075093 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.075598 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.076583 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.077011 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.077084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.078800 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.078802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.078802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:33.078809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.080300 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:33.080664 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:33.080665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:33.080665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:33.080726 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:33.080953 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:33.080954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:33.080954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:33.081126 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.081226 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.081434 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.081565 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:28:33.083420 12030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:33.085093 12030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:33.092772 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39895 (local address 127.0.0.1:36464)
0928 20:28:33.087888 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.087955 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.088057 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.088450 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.088453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.088456 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.088551 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.088554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.089883 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.089884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.091268 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.091272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092399 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.092489 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092508 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.092585 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.092653 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":150,"threads_started":1}
I20260928 20:28:33.093039 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:34269 (local address 127.0.0.1:56994)
0928 20:28:33.088195 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.088206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.088514 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.088774 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.088775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.088777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.088823 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.088825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.090119 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.090120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.091427 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.091437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092793 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.092864 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092869 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.092921 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.092968 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":101,"threads_started":1}
I20260928 20:28:33.093122 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.24.186.193:39895)
0928 20:28:33.088253 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.088258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.088268 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:28:33.088308 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.088310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.088328 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.088410 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.088651 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.089756 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.091381 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092263 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092900 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092959 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.092987 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.093026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260928 20:28:33.093329 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:41993 (local address 127.0.0.1:41170)
0928 20:28:33.088346 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.088357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.088396 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.089098 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.089099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.089101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.089138 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.089139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.090360 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.090361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.091625 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.091628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092610 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.092691 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092697 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.092770 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.093031 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260928 20:28:33.093427 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.24.186.194:41993)
0928 20:28:33.088886 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.088890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.088899 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:33.088934 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.088937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.088952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.089030 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.089214 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.090258 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.091750 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092485 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092567 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092627 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.093294 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.093343 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":833,"threads_started":1}
I20260928 20:28:33.093909 12155 tablet_service.cc:1467] Processing CreateTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 (DEFAULT_TABLE table=test-table [id=d2f984a81a554cb9a5ef6ea240953f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.094295 12155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454e2da27ecc41cca58f9d642b1d4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.094295 12289 tablet_service.cc:1467] Processing CreateTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 (DEFAULT_TABLE table=test-table [id=d2f984a81a554cb9a5ef6ea240953f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.094602 12289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454e2da27ecc41cca58f9d642b1d4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.097411 12508 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Bootstrap starting.
I20260928 20:28:33.097640 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.087951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.24.186.195:34269)
0928 20:28:33.088136 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.088141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.088276 (+ 135us) server_negotiation.cc:407] Connection header received
0928 20:28:33.088619 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.088623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.088639 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.088708 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.088904 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.090012 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.091694 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.092693 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.092784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.092845 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.097441 (+ 4596us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.097524 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260928 20:28:33.098266 12508 tablet_bootstrap.cc:654] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.098275 12423 tablet_service.cc:1467] Processing CreateTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 (DEFAULT_TABLE table=test-table [id=d2f984a81a554cb9a5ef6ea240953f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.098395 12507 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Bootstrap starting.
I20260928 20:28:33.098625 12423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454e2da27ecc41cca58f9d642b1d4329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.099174 12507 tablet_bootstrap.cc:654] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.099339 12508 log.cc:824] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.099937 12507 log.cc:824] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.100555 12508 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: No bootstrap required, opened a new log
I20260928 20:28:33.100626 12508 ts_tablet_manager.cc:1397] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:28:33.102211 12507 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: No bootstrap required, opened a new log
I20260928 20:28:33.102330 12507 ts_tablet_manager.cc:1397] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 20:28:33.103042 12508 raft_consensus.cc:348] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.103245 12508 raft_consensus.cc:374] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.103292 12508 raft_consensus.cc:729] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c899c2d05864382b8ef0eef00c53913, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.103545 12508 consensus_queue.cc:260] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.103971 12511 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Bootstrap starting.
I20260928 20:28:33.103998 12508 ts_tablet_manager.cc:1428] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 20:28:33.104130 12355 heartbeater.cc:503] Master 127.24.186.254:43257 was elected leader, sending a full tablet report...
I20260928 20:28:33.104610 12507 raft_consensus.cc:348] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.104789 12507 raft_consensus.cc:374] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.104825 12507 raft_consensus.cc:729] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943997bf31914ebd88175e16ebd6a5f1, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.104923 12507 consensus_queue.cc:260] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.104959 12511 tablet_bootstrap.cc:654] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.105196 12507 ts_tablet_manager.cc:1428] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:28:33.105353 12221 heartbeater.cc:503] Master 127.24.186.254:43257 was elected leader, sending a full tablet report...
I20260928 20:28:33.105556 12511 log.cc:824] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.106433 12511 tablet_bootstrap.cc:492] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: No bootstrap required, opened a new log
I20260928 20:28:33.106500 12511 ts_tablet_manager.cc:1397] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 20:28:33.108690 12511 raft_consensus.cc:348] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.108886 12511 raft_consensus.cc:374] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.108928 12511 raft_consensus.cc:729] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0288d4b514134ceb958cc40ccb17a9ba, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.109055 12511 consensus_queue.cc:260] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.109338 12511 ts_tablet_manager.cc:1428] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 20:28:33.109366 12490 heartbeater.cc:503] Master 127.24.186.254:43257 was elected leader, sending a full tablet report...
W20260928 20:28:33.129988 12356 tablet.cc:2369] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 20:28:33.202175 12222 tablet.cc:2369] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:33.247104 12513 raft_consensus.cc:482] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:33.247231 12513 raft_consensus.cc:504] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.247579 12513 leader_election.cc:288] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0288d4b514134ceb958cc40ccb17a9ba (127.24.186.195:34269), 5c899c2d05864382b8ef0eef00c53913 (127.24.186.194:41993)
I20260928 20:28:33.250849 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.247928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:57741 (local address 127.24.186.195:34269)
0928 20:28:33.247976 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.247978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.248181 (+ 203us) server_negotiation.cc:407] Connection header received
0928 20:28:33.248264 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.248265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.248274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.248359 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.248529 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.249075 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250105 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.250489 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250580 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.250589 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.250736 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.250770 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:28:33.250861 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.247785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:34269 (local address 127.24.186.193:57741)
0928 20:28:33.248096 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.248106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.248191 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.248366 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.248367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.248369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.248454 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.248456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.249159 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.249160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.250018 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.250021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.250580 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.250651 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.250657 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.250731 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.250769 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260928 20:28:33.251030 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.247855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:58263 (local address 127.24.186.194:41993)
0928 20:28:33.247922 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.247924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.248077 (+ 153us) server_negotiation.cc:407] Connection header received
0928 20:28:33.248154 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.248155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.248167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.248236 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.248450 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.249045 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250190 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.250681 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250754 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.250763 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.250926 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.250962 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 20:28:33.251048 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.247767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:41993 (local address 127.24.186.193:58263)
0928 20:28:33.247977 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.247989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.248094 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.248240 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.248241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.248243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.248333 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.248335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.249140 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.249141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.250104 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.250106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.250756 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.250757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.250830 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.250834 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.250918 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.250961 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":26,"thread_start_us":145,"threads_started":1}
I20260928 20:28:33.251322 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454e2da27ecc41cca58f9d642b1d4329" candidate_uuid: "943997bf31914ebd88175e16ebd6a5f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0288d4b514134ceb958cc40ccb17a9ba" is_pre_election: true
I20260928 20:28:33.251489 12443 raft_consensus.cc:2454] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943997bf31914ebd88175e16ebd6a5f1 in term 0.
I20260928 20:28:33.251546 12309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454e2da27ecc41cca58f9d642b1d4329" candidate_uuid: "943997bf31914ebd88175e16ebd6a5f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c899c2d05864382b8ef0eef00c53913" is_pre_election: true
I20260928 20:28:33.251713 12309 raft_consensus.cc:2454] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 943997bf31914ebd88175e16ebd6a5f1 in term 0.
I20260928 20:28:33.251700 12110 leader_election.cc:302] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0288d4b514134ceb958cc40ccb17a9ba, 943997bf31914ebd88175e16ebd6a5f1; no voters:
I20260928 20:28:33.251843 12513 raft_consensus.cc:2781] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:33.251904 12513 raft_consensus.cc:482] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:33.251930 12513 raft_consensus.cc:3037] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:33.252625 12513 raft_consensus.cc:504] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.252776 12513 leader_election.cc:288] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [CANDIDATE]: Term 1 election: Requested vote from peers 0288d4b514134ceb958cc40ccb17a9ba (127.24.186.195:34269), 5c899c2d05864382b8ef0eef00c53913 (127.24.186.194:41993)
I20260928 20:28:33.252943 12309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454e2da27ecc41cca58f9d642b1d4329" candidate_uuid: "943997bf31914ebd88175e16ebd6a5f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c899c2d05864382b8ef0eef00c53913"
I20260928 20:28:33.252943 12443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "454e2da27ecc41cca58f9d642b1d4329" candidate_uuid: "943997bf31914ebd88175e16ebd6a5f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0288d4b514134ceb958cc40ccb17a9ba"
I20260928 20:28:33.253021 12309 raft_consensus.cc:3037] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:33.253026 12443 raft_consensus.cc:3037] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:33.253933 12309 raft_consensus.cc:2454] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943997bf31914ebd88175e16ebd6a5f1 in term 1.
I20260928 20:28:33.253933 12443 raft_consensus.cc:2454] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 943997bf31914ebd88175e16ebd6a5f1 in term 1.
I20260928 20:28:33.254110 12108 leader_election.cc:302] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c899c2d05864382b8ef0eef00c53913, 943997bf31914ebd88175e16ebd6a5f1; no voters:
I20260928 20:28:33.254223 12513 raft_consensus.cc:2781] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:33.254369 12513 raft_consensus.cc:686] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 1 LEADER]: Becoming Leader. State: Replica: 943997bf31914ebd88175e16ebd6a5f1, State: Running, Role: LEADER
I20260928 20:28:33.254477 12513 consensus_queue.cc:237] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } }
I20260928 20:28:33.255352 12030 catalog_manager.cc:5495] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 943997bf31914ebd88175e16ebd6a5f1 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "943997bf31914ebd88175e16ebd6a5f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "943997bf31914ebd88175e16ebd6a5f1" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 39895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 } health_report { overall_health: UNKNOWN } } }
I20260928 20:28:33.279358 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.276715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.24.186.193:39895)
0928 20:28:33.276798 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.276800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.276809 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:28:33.276857 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.276858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.276877 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.276954 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:33.277103 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.277704 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.278519 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.278713 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.278789 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.278985 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:33.279102 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:33.279199 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.279222 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.279247 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 20:28:33.281342 12309 raft_consensus.cc:1261] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 1 FOLLOWER]: Refusing update from remote peer 943997bf31914ebd88175e16ebd6a5f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:33.281358 12443 raft_consensus.cc:1261] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 1 FOLLOWER]: Refusing update from remote peer 943997bf31914ebd88175e16ebd6a5f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:33.281741 12513 consensus_queue.cc:1035] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c899c2d05864382b8ef0eef00c53913" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:33.281906 12518 consensus_queue.cc:1035] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0288d4b514134ceb958cc40ccb17a9ba" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:33.288008 12029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:33.288121 12029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:33.289016 12029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb228541119842879bab829c1d1e83b1: Sending DeleteTablet for 3 replicas of tablet 454e2da27ecc41cca58f9d642b1d4329
I20260928 20:28:33.289417 12155 tablet_service.cc:1514] Processing DeleteTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:36464
I20260928 20:28:33.289443 12289 tablet_service.cc:1514] Processing DeleteTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:41170
I20260928 20:28:33.289705 12530 tablet_replica.cc:330] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1: stopping tablet replica
I20260928 20:28:33.289733 12423 tablet_service.cc:1514] Processing DeleteTablet for tablet 454e2da27ecc41cca58f9d642b1d4329 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:33 UTC) from {username='kudu', principal='kudu/127.24.186.254@KRBTEST.COM'} at 127.0.0.1:56994
I20260928 20:28:33.289816 12531 tablet_replica.cc:330] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: stopping tablet replica
I20260928 20:28:33.289858 12530 raft_consensus.cc:2229] T 454e2da27ecc41cca58f9d642b1d4329 P 943997bf31914ebd88175e16ebd6a5f1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:28:33.289949 12531 raft_consensus.cc:2229] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:33.290140 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12092
I20260928 20:28:33.290107 12531 raft_consensus.cc:2258] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:33.291348 12532 tablet_replica.cc:330] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: stopping tablet replica
I20260928 20:28:33.291534 12532 raft_consensus.cc:2229] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:33.291599 12531 ts_tablet_manager.cc:1905] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:33.291695 12532 raft_consensus.cc:2258] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:33.292225 12532 ts_tablet_manager.cc:1905] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:33.293737 12531 ts_tablet_manager.cc:1918] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:33.293823 12531 log.cc:1196] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/454e2da27ecc41cca58f9d642b1d4329
I20260928 20:28:33.294121 12531 ts_tablet_manager.cc:1939] T 454e2da27ecc41cca58f9d642b1d4329 P 5c899c2d05864382b8ef0eef00c53913: Deleting consensus metadata
I20260928 20:28:33.294209 12532 ts_tablet_manager.cc:1918] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:33.294277 12532 log.cc:1196] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/454e2da27ecc41cca58f9d642b1d4329
I20260928 20:28:33.294517 12532 ts_tablet_manager.cc:1939] T 454e2da27ecc41cca58f9d642b1d4329 P 0288d4b514134ceb958cc40ccb17a9ba: Deleting consensus metadata
I20260928 20:28:33.295018 12014 catalog_manager.cc:4841] TS 5c899c2d05864382b8ef0eef00c53913 (127.24.186.194:41993): tablet 454e2da27ecc41cca58f9d642b1d4329 (table test-table [id=d2f984a81a554cb9a5ef6ea240953f32]) successfully deleted
I20260928 20:28:33.295092 12016 catalog_manager.cc:4841] TS 0288d4b514134ceb958cc40ccb17a9ba (127.24.186.195:34269): tablet 454e2da27ecc41cca58f9d642b1d4329 (table test-table [id=d2f984a81a554cb9a5ef6ea240953f32]) successfully deleted
W20260928 20:28:33.296671 12017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:39895 (error 108)
I20260928 20:28:33.296932 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12226
I20260928 20:28:33.297909 12501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.297130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:39895 (local address 127.0.0.1:36492)
0928 20:28:33.297832 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.297875 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681}
W20260928 20:28:33.298084 12017 catalog_manager.cc:4584] TS 943997bf31914ebd88175e16ebd6a5f1 (127.24.186.193:39895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 454e2da27ecc41cca58f9d642b1d4329: Network error: Client connection negotiation failed: client connection to 127.24.186.193:39895: connect: Connection refused (error 111)
I20260928 20:28:33.303093 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12360
I20260928 20:28:33.307739 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 11998
2026-09-28T20:28:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2151 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T20:28:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:33Z Disabled control of system clock
I20260928 20:28:33.337808 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:36205
--webserver_interface=127.24.186.254
--webserver_port=0
--builtin_ntp_servers=127.24.186.212:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:33.421380 12537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:33.421636 12537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:33.421659 12537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:33.423031 12537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:33.423082 12537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:33.423095 12537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:33.423108 12537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:33.423161 12537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:33.424573 12537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:36205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:33.424853 12537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:33.425163 12537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:33.430037 12545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.430091 12542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.430332 12537 server_base.cc:1034] running on GCE node
W20260928 20:28:33.430039 12543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.431712 12537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:33.432049 12537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:33.433243 12537 hybrid_clock.cc:648] HybridClock initialized: now 1790627313433226 us; error 44 us; skew 500 ppm
I20260928 20:28:33.434633 12537 webserver.cc:460] Webserver started at http://127.24.186.254:44755/ using document root <none> and password file <none>
I20260928 20:28:33.434890 12537 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:33.434938 12537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:33.435092 12537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:33.436280 12537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "09e12ef850734c5c83f3dc829d46aea5"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "b8f9ec000b4ccb26a68937c8675ca249"
server_key_iv: "94f41875de156c65ed4c7ba4293b7127"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.436650 12537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "09e12ef850734c5c83f3dc829d46aea5"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "b8f9ec000b4ccb26a68937c8675ca249"
server_key_iv: "94f41875de156c65ed4c7ba4293b7127"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.437975 12537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:33.438727 12552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.438921 12537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 20:28:33.439013 12537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "09e12ef850734c5c83f3dc829d46aea5"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "b8f9ec000b4ccb26a68937c8675ca249"
server_key_iv: "94f41875de156c65ed4c7ba4293b7127"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.439088 12537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:33.452149 12537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:33.452513 12537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:33.452690 12537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:33.456758 12537 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:36205
I20260928 20:28:33.456827 12604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:36205 every 8 connection(s)
I20260928 20:28:33.457173 12537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:33.458266 12605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.460636 12605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: Bootstrap starting.
I20260928 20:28:33.461221 12605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.461529 12605 log.cc:824] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.462242 12605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: No bootstrap required, opened a new log
I20260928 20:28:33.463204 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12537
I20260928 20:28:33.463290 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:33.463500 25323 external_mini_cluster.cc:1428] Setting key 92d3c62a2166e10c8ca31de24d768863
I20260928 20:28:33.464279 12605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } }
I20260928 20:28:33.464461 12605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.464496 12605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e12ef850734c5c83f3dc829d46aea5, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.464620 12605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } }
I20260928 20:28:33.464708 12605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:33.464740 12605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:33.464784 12605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:33.465565 12605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } }
I20260928 20:28:33.465672 12605 leader_election.cc:302] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e12ef850734c5c83f3dc829d46aea5; no voters:
I20260928 20:28:33.465840 12605 leader_election.cc:288] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:33.465919 12610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:33.466073 12610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [term 1 LEADER]: Becoming Leader. State: Replica: 09e12ef850734c5c83f3dc829d46aea5, State: Running, Role: LEADER
I20260928 20:28:33.466081 12605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:33.466171 12610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } }
I20260928 20:28:33.466506 12611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e12ef850734c5c83f3dc829d46aea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } } }
I20260928 20:28:33.466604 12611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:33.466497 12610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e12ef850734c5c83f3dc829d46aea5. Latest consensus state: current_term: 1 leader_uuid: "09e12ef850734c5c83f3dc829d46aea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e12ef850734c5c83f3dc829d46aea5" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 36205 } } }
I20260928 20:28:33.466806 12610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:33.467015 12617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:33.467545 12617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:33.469517 12617 catalog_manager.cc:1347] Generated new cluster ID: 56d744629b7f47bdb6363a7507f5a1ca
I20260928 20:28:33.469573 12617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:33.470041 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.464068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.24.186.254:36205)
0928 20:28:33.464349 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.464358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.464390 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:28:33.464501 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.464509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.464578 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.464739 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.464978 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.466791 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.468075 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.468481 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.468616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.469331 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.469335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.469336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:33.469590 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.469637 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:33.469653 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.469711 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.469785 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.469865 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":136,"threads_started":1}
I20260928 20:28:33.495399 12617 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:33.496142 12617 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:33.505002 12617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: Generated new TSK 0
I20260928 20:28:33.505292 12617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:33.514995 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.193:0
--local_ip_for_outbound_sockets=127.24.186.193
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--builtin_ntp_servers=127.24.186.212:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 20:28:33.612155 12629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:33.612406 12629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:33.612440 12629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:33.613638 12629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:33.613709 12629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:33.613754 12629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.193
I20260928 20:28:33.615146 12629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.24.186.193
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.193
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:33.615420 12629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:33.615743 12629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:33.622179 12634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.622185 12635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.622185 12637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.622466 12629 server_base.cc:1034] running on GCE node
I20260928 20:28:33.622699 12629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:33.623023 12629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:33.624195 12629 hybrid_clock.cc:648] HybridClock initialized: now 1790627313624174 us; error 34 us; skew 500 ppm
I20260928 20:28:33.625604 12629 webserver.cc:460] Webserver started at http://127.24.186.193:35045/ using document root <none> and password file <none>
I20260928 20:28:33.625906 12629 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:33.625984 12629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:33.626157 12629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:33.627300 12629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data/instance:
uuid: "b3adb614ffe84e47aa32619987d9b626"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "5977c6af193c4550a936e4bf0ad69bb3"
server_key_iv: "39079e353ef01f4c46f519762f1bc58a"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.627674 12629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance:
uuid: "b3adb614ffe84e47aa32619987d9b626"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "5977c6af193c4550a936e4bf0ad69bb3"
server_key_iv: "39079e353ef01f4c46f519762f1bc58a"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.629755 12629 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 20:28:33.630626 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.630801 12629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:28:33.630899 12629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal
uuid: "b3adb614ffe84e47aa32619987d9b626"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "5977c6af193c4550a936e4bf0ad69bb3"
server_key_iv: "39079e353ef01f4c46f519762f1bc58a"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.630959 12629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:33.647981 12629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:33.648274 12629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:33.648384 12629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:33.648588 12629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:33.648962 12629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:33.648998 12629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.649055 12629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:33.649081 12629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.655004 12629 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.193:36369
I20260928 20:28:33.655133 12757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.193:36369 every 8 connection(s)
I20260928 20:28:33.655469 12629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/data/info.pb
I20260928 20:28:33.661238 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.656521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:44637 (local address 127.24.186.254:36205)
0928 20:28:33.656656 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.656658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.656855 (+ 197us) server_negotiation.cc:407] Connection header received
0928 20:28:33.657419 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.657420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.657435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.657499 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.658191 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.658702 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.659691 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.659958 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.660037 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.660918 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.660920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.660920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:33.660928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.660944 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:33.660949 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.661023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.661142 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.661173 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 20:28:33.661279 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12629
I20260928 20:28:33.661386 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-0/wal/instance
I20260928 20:28:33.661502 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.656487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36205 (local address 127.24.186.193:44637)
0928 20:28:33.656732 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.656767 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.657311 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.657571 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.657580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.657589 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:33.658087 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.658091 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.658808 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.658809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.659590 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.659592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.660046 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.660047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.660187 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.660765 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:33.660765 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:33.660793 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:33.660815 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:33.661023 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:33.661030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:33.661033 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.661125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.661248 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260928 20:28:33.661625 25323 external_mini_cluster.cc:1428] Setting key 735dec8533166f7a831cce9520fcb199
I20260928 20:28:33.662055 12758 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36205
I20260928 20:28:33.662155 12758 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:33.662331 12758 heartbeater.cc:511] Master 127.24.186.254:36205 requested a full tablet report, sending...
I20260928 20:28:33.662885 12569 ts_manager.cc:194] Registered new tserver with Master: b3adb614ffe84e47aa32619987d9b626 (127.24.186.193:36369)
I20260928 20:28:33.662971 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.194:0
--local_ip_for_outbound_sockets=127.24.186.194
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--builtin_ntp_servers=127.24.186.212:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 20:28:33.663848 12569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.193:44637
W20260928 20:28:33.749590 12761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:33.749876 12761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:33.749915 12761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:33.751149 12761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:33.751219 12761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:33.751262 12761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.194
I20260928 20:28:33.752658 12761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.24.186.194
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.194
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:33.752935 12761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:33.753247 12761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:33.758307 12766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.758343 12769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.758522 12767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.758626 12761 server_base.cc:1034] running on GCE node
I20260928 20:28:33.758853 12761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:33.759066 12761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:33.760208 12761 hybrid_clock.cc:648] HybridClock initialized: now 1790627313760199 us; error 30 us; skew 500 ppm
I20260928 20:28:33.761732 12761 webserver.cc:460] Webserver started at http://127.24.186.194:39173/ using document root <none> and password file <none>
I20260928 20:28:33.761996 12761 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:33.762046 12761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:33.762212 12761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:33.763356 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data/instance:
uuid: "4eec1d9c0d594bf08ec53f8e3f16db05"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "453d0c6745a4430440a714fd0a2e0e82"
server_key_iv: "3f1364e308ebb313685c87713be980a6"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.763725 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance:
uuid: "4eec1d9c0d594bf08ec53f8e3f16db05"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "453d0c6745a4430440a714fd0a2e0e82"
server_key_iv: "3f1364e308ebb313685c87713be980a6"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.765168 12761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:33.766036 12775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.766356 12761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:33.766459 12761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal
uuid: "4eec1d9c0d594bf08ec53f8e3f16db05"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "453d0c6745a4430440a714fd0a2e0e82"
server_key_iv: "3f1364e308ebb313685c87713be980a6"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.766541 12761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:33.785959 12761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:33.786336 12761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:33.786501 12761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:33.786765 12761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:33.787134 12761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:33.787189 12761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.787226 12761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:33.787387 12761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.795089 12761 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.194:34255
I20260928 20:28:33.795171 12889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.194:34255 every 8 connection(s)
I20260928 20:28:33.795492 12761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/data/info.pb
I20260928 20:28:33.798974 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12761
I20260928 20:28:33.799085 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal/instance
I20260928 20:28:33.799319 25323 external_mini_cluster.cc:1428] Setting key 6f17264d6f8e692e6a8d3ed7200424a8
I20260928 20:28:33.800400 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.24.186.195:0
--local_ip_for_outbound_sockets=127.24.186.195
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--builtin_ntp_servers=127.24.186.212:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 20:28:33.801352 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.796266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.194:57725 (local address 127.24.186.254:36205)
0928 20:28:33.796345 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.796347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.796622 (+ 275us) server_negotiation.cc:407] Connection header received
0928 20:28:33.797131 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.797133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.797143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.797210 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.797940 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.798455 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.799742 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.800038 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.800123 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.801036 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.801038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.801038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:33.801044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.801064 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:33.801069 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.801133 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.801197 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.801232 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:28:33.801504 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.796238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36205 (local address 127.24.186.194:57725)
0928 20:28:33.796460 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.796501 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.797010 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.797284 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.797288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.797296 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:33.797825 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.797829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.798593 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.798594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.799635 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.799639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.800118 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.800119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.800266 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.800893 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:33.800894 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:33.800924 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:33.800948 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:33.801139 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:33.801143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:33.801145 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.801197 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.801287 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
I20260928 20:28:33.802229 12890 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36205
I20260928 20:28:33.802336 12890 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:33.802526 12890 heartbeater.cc:511] Master 127.24.186.254:36205 requested a full tablet report, sending...
I20260928 20:28:33.802965 12569 ts_manager.cc:194] Registered new tserver with Master: 4eec1d9c0d594bf08ec53f8e3f16db05 (127.24.186.194:34255)
I20260928 20:28:33.803339 12569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.194:57725
W20260928 20:28:33.892424 12893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:33.892704 12893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:33.892730 12893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:33.894210 12893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:33.894261 12893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:33.894290 12893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.186.195
I20260928 20:28:33.895716 12893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.24.186.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.24.186.195
--webserver_port=0
--tserver_master_addrs=127.24.186.254:36205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.24.186.195
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:33.896027 12893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:33.896325 12893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:33.901890 12898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.901926 12893 server_base.cc:1034] running on GCE node
W20260928 20:28:33.901988 12901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:33.901919 12899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:33.902426 12893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:33.902658 12893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:33.903868 12893 hybrid_clock.cc:648] HybridClock initialized: now 1790627313903840 us; error 41 us; skew 500 ppm
I20260928 20:28:33.905898 12893 webserver.cc:460] Webserver started at http://127.24.186.195:41853/ using document root <none> and password file <none>
I20260928 20:28:33.906352 12893 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:33.906411 12893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:33.906518 12893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:33.907615 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data/instance:
uuid: "ab0d091856b445dd88b712632863d596"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "fb7da7acdd996f144cf5233d5d6075e1"
server_key_iv: "8b33aa6a69b624e2862a2bb9c60a7101"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.908028 12893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance:
uuid: "ab0d091856b445dd88b712632863d596"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "fb7da7acdd996f144cf5233d5d6075e1"
server_key_iv: "8b33aa6a69b624e2862a2bb9c60a7101"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.909507 12893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:33.910588 12907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.911000 12893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:33.911139 12893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal
uuid: "ab0d091856b445dd88b712632863d596"
format_stamp: "Formatted at 2026-09-28 20:28:33 on dist-test-slave-k477"
server_key: "fb7da7acdd996f144cf5233d5d6075e1"
server_key_iv: "8b33aa6a69b624e2862a2bb9c60a7101"
server_key_version: "encryptionkey@0"
I20260928 20:28:33.911376 12893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:33.933144 12893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:33.933523 12893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:33.933694 12893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:33.934278 12893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 20:28:33.934707 12893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 20:28:33.934743 12893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.934799 12893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 20:28:33.934827 12893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:33.941587 12893 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.195:45917
I20260928 20:28:33.941627 13020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.195:45917 every 8 connection(s)
I20260928 20:28:33.942054 12893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/data/info.pb
I20260928 20:28:33.947343 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 12893
I20260928 20:28:33.947496 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal/instance
I20260928 20:28:33.947701 25323 external_mini_cluster.cc:1428] Setting key d1578d86f7b3453e66df0917774a5fcb
I20260928 20:28:33.949090 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.942834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.195:52253 (local address 127.24.186.254:36205)
0928 20:28:33.942883 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.942885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.943155 (+ 270us) server_negotiation.cc:407] Connection header received
0928 20:28:33.943776 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.943778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.943789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.943946 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.944836 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.945466 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.947345 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.947625 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.947713 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.948804 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.948806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.948806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:33.948812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.948826 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:33.948830 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.948888 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.948961 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.948995 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:28:33.949340 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.942803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:36205 (local address 127.24.186.195:52253)
0928 20:28:33.943004 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.943055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.943592 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.944016 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.944020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.944028 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:28:33.944707 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.944713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.945986 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.945988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.947174 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.947180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.947725 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.947726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.947972 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.948653 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 20:28:33.948653 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:28:33.948685 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 20:28:33.948712 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:28:33.948891 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:28:33.948896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:28:33.948900 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.948968 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.949090 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260928 20:28:33.949783 13021 heartbeater.cc:347] Connected to a master server at 127.24.186.254:36205
I20260928 20:28:33.949875 13021 heartbeater.cc:464] Registering TS with master...
I20260928 20:28:33.950096 13021 heartbeater.cc:511] Master 127.24.186.254:36205 requested a full tablet report, sending...
I20260928 20:28:33.950663 12569 ts_manager.cc:194] Registered new tserver with Master: ab0d091856b445dd88b712632863d596 (127.24.186.195:45917)
I20260928 20:28:33.951114 12569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.24.186.195:52253
I20260928 20:28:33.960121 25323 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 20:28:33.965127 12609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.961730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.24.186.254:36205)
0928 20:28:33.961821 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.961823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.961924 (+ 101us) server_negotiation.cc:407] Connection header received
0928 20:28:33.962025 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.962026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.962040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.962118 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:33.962307 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.962816 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.963584 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.963788 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.963923 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.964768 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:33.964770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:33.964770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:33.964777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:33.964797 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:33.964807 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:33.964880 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.964979 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.965011 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 20:28:33.967201 12569 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 20:28:33.970353 12569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 20:28:33.978123 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41430 (local address 127.24.186.195:45917)
0928 20:28:33.973954 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.973957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.973965 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:28:33.974001 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.974003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.974017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.974445 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.974509 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975344 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977049 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977762 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977831 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.977906 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.977997 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.978034 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
I20260928 20:28:33.978119 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45917 (local address 127.0.0.1:41430)
0928 20:28:33.973618 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.973659 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.973730 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.974145 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.974148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.974151 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.974333 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.974340 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975453 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.975455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.976944 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.976947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977831 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.977907 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.977914 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.977976 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.978034 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260928 20:28:33.978132 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.24.186.194:34255)
0928 20:28:33.974056 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.974061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.974238 (+ 177us) server_negotiation.cc:407] Connection header received
0928 20:28:33.974282 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.974284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.974299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.974367 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.974555 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975649 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.976828 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977534 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977637 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.977753 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.977999 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.978047 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":136,"threads_started":1}
I20260928 20:28:33.978042 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36369 (local address 127.0.0.1:37924)
0928 20:28:33.973836 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.973845 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.973882 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.974215 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.974216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.974220 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.974290 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.974293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975725 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.975726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.976957 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.976959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977792 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.977864 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.977866 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.977898 (+ 32us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.977930 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
I20260928 20:28:33.978392 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:34255 (local address 127.0.0.1:52108)
0928 20:28:33.974076 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:33.974085 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:33.974177 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:33.974433 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:33.974434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:33.974436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:33.974466 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.974467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975736 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.975737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.976740 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:33.976743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977647 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:33.977767 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.977789 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 20:28:33.977864 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 20:28:33.977930 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":40,"threads_started":1}
I20260928 20:28:33.978937 12955 tablet_service.cc:1467] Processing CreateTablet for tablet e6020337bc954d2e949692abd87efbca (DEFAULT_TABLE table=test-table [id=cd6e69ae0e4a436bb09affca1e5e3c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.979338 12955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6020337bc954d2e949692abd87efbca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.979327 12824 tablet_service.cc:1467] Processing CreateTablet for tablet e6020337bc954d2e949692abd87efbca (DEFAULT_TABLE table=test-table [id=cd6e69ae0e4a436bb09affca1e5e3c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.979691 12824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6020337bc954d2e949692abd87efbca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.982205 13035 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Bootstrap starting.
I20260928 20:28:33.982790 13035 tablet_bootstrap.cc:654] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.982806 13036 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Bootstrap starting.
I20260928 20:28:33.983112 13035 log.cc:824] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.983263 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:33.973737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.24.186.193:36369)
0928 20:28:33.973920 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:33.973925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:33.973938 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:28:33.973985 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:33.973987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:33.974004 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:33.974132 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:33.974385 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.975626 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.977067 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:33.977673 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:33.978092 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:33.983027 (+ 4935us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:33.983069 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 20:28:33.983150 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260928 20:28:33.983523 13036 tablet_bootstrap.cc:654] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.983911 13036 log.cc:824] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.984045 12692 tablet_service.cc:1467] Processing CreateTablet for tablet e6020337bc954d2e949692abd87efbca (DEFAULT_TABLE table=test-table [id=cd6e69ae0e4a436bb09affca1e5e3c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 20:28:33.984380 12692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6020337bc954d2e949692abd87efbca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:33.984704 13036 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: No bootstrap required, opened a new log
I20260928 20:28:33.984766 13036 ts_tablet_manager.cc:1397] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:28:33.987133 13039 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: Bootstrap starting.
I20260928 20:28:33.987139 13036 raft_consensus.cc:348] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.987363 13036 raft_consensus.cc:374] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.987413 13036 raft_consensus.cc:729] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eec1d9c0d594bf08ec53f8e3f16db05, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.987552 13036 consensus_queue.cc:260] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.987870 13036 ts_tablet_manager.cc:1428] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:28:33.987900 12890 heartbeater.cc:503] Master 127.24.186.254:36205 was elected leader, sending a full tablet report...
I20260928 20:28:33.987921 13039 tablet_bootstrap.cc:654] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:33.987938 13035 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: No bootstrap required, opened a new log
I20260928 20:28:33.987999 13035 ts_tablet_manager.cc:1397] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 20:28:33.988245 13039 log.cc:824] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:33.989061 13039 tablet_bootstrap.cc:492] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: No bootstrap required, opened a new log
I20260928 20:28:33.989125 13039 ts_tablet_manager.cc:1397] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 20:28:33.990746 13035 raft_consensus.cc:348] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.990937 13035 raft_consensus.cc:374] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.990972 13035 raft_consensus.cc:729] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab0d091856b445dd88b712632863d596, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.991092 13035 consensus_queue.cc:260] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.991374 13035 ts_tablet_manager.cc:1428] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 20:28:33.991426 13021 heartbeater.cc:503] Master 127.24.186.254:36205 was elected leader, sending a full tablet report...
I20260928 20:28:33.991724 13039 raft_consensus.cc:348] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.991923 13039 raft_consensus.cc:374] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:33.991969 13039 raft_consensus.cc:729] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3adb614ffe84e47aa32619987d9b626, State: Initialized, Role: FOLLOWER
I20260928 20:28:33.992095 13039 consensus_queue.cc:260] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:33.992391 13039 ts_tablet_manager.cc:1428] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 20:28:33.992475 12758 heartbeater.cc:503] Master 127.24.186.254:36205 was elected leader, sending a full tablet report...
W20260928 20:28:34.046183 12891 tablet.cc:2369] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:28:34.075129 13043 raft_consensus.cc:482] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 20:28:34.075271 13043 raft_consensus.cc:504] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:34.075642 13043 leader_election.cc:288] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab0d091856b445dd88b712632863d596 (127.24.186.195:45917), 4eec1d9c0d594bf08ec53f8e3f16db05 (127.24.186.194:34255)
I20260928 20:28:34.078872 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.076014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:38287 (local address 127.24.186.195:45917)
0928 20:28:34.076070 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:34.076072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:34.076083 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:28:34.076117 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:34.076118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:34.076128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:34.076225 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:34.076582 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.077066 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078110 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.078489 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078590 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:34.078598 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:34.078746 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 20:28:34.078795 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 20:28:34.078871 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.076107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.24.186.193:42533 (local address 127.24.186.194:34255)
0928 20:28:34.076166 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:34.076168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:34.076222 (+ 54us) server_negotiation.cc:407] Connection header received
0928 20:28:34.076293 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:34.076294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:34.076303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:34.076378 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 20:28:34.076606 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.077073 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078124 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.078495 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078582 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:34.078592 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:34.078760 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 20:28:34.078798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 20:28:34.078886 12760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.075866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.195:45917 (local address 127.24.186.193:38287)
0928 20:28:34.075940 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:34.075955 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:34.076024 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:34.076231 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:34.076232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:34.076234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:34.076491 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:34.076494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.077168 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.077170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:34.078008 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:34.078010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.078591 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:34.078667 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:34.078670 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:34.078738 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:28:34.078787 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":79}
I20260928 20:28:34.078902 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.075983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.194:34255 (local address 127.24.186.193:42533)
0928 20:28:34.076169 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:34.076178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:28:34.076229 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:28:34.076384 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:28:34.076384 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:28:34.076386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 20:28:34.076497 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:34.076498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.077162 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.077163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:34.078020 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:28:34.078022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.078601 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.078602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:28:34.078683 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:34.078686 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 20:28:34.078753 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 20:28:34.078793 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":33,"thread_start_us":120,"threads_started":1}
I20260928 20:28:34.079291 12975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6020337bc954d2e949692abd87efbca" candidate_uuid: "b3adb614ffe84e47aa32619987d9b626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0d091856b445dd88b712632863d596" is_pre_election: true
I20260928 20:28:34.079283 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6020337bc954d2e949692abd87efbca" candidate_uuid: "b3adb614ffe84e47aa32619987d9b626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" is_pre_election: true
I20260928 20:28:34.079453 12844 raft_consensus.cc:2454] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3adb614ffe84e47aa32619987d9b626 in term 0.
I20260928 20:28:34.079453 12975 raft_consensus.cc:2454] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3adb614ffe84e47aa32619987d9b626 in term 0.
I20260928 20:28:34.079690 12646 leader_election.cc:302] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eec1d9c0d594bf08ec53f8e3f16db05, b3adb614ffe84e47aa32619987d9b626; no voters:
I20260928 20:28:34.079846 13043 raft_consensus.cc:2781] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 20:28:34.079936 13043 raft_consensus.cc:482] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 20:28:34.079974 13043 raft_consensus.cc:3037] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:34.080724 13043 raft_consensus.cc:504] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:34.080881 13043 leader_election.cc:288] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [CANDIDATE]: Term 1 election: Requested vote from peers ab0d091856b445dd88b712632863d596 (127.24.186.195:45917), 4eec1d9c0d594bf08ec53f8e3f16db05 (127.24.186.194:34255)
I20260928 20:28:34.081050 12975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6020337bc954d2e949692abd87efbca" candidate_uuid: "b3adb614ffe84e47aa32619987d9b626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab0d091856b445dd88b712632863d596"
I20260928 20:28:34.081128 12975 raft_consensus.cc:3037] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:34.081130 12844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6020337bc954d2e949692abd87efbca" candidate_uuid: "b3adb614ffe84e47aa32619987d9b626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05"
I20260928 20:28:34.081220 12844 raft_consensus.cc:3037] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:34.081967 12975 raft_consensus.cc:2454] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3adb614ffe84e47aa32619987d9b626 in term 1.
I20260928 20:28:34.082131 12844 raft_consensus.cc:2454] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3adb614ffe84e47aa32619987d9b626 in term 1.
I20260928 20:28:34.082161 12646 leader_election.cc:302] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab0d091856b445dd88b712632863d596, b3adb614ffe84e47aa32619987d9b626; no voters:
I20260928 20:28:34.082315 13043 raft_consensus.cc:2781] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:34.082500 13043 raft_consensus.cc:686] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 1 LEADER]: Becoming Leader. State: Replica: b3adb614ffe84e47aa32619987d9b626, State: Running, Role: LEADER
I20260928 20:28:34.082732 13043 consensus_queue.cc:237] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } }
I20260928 20:28:34.083751 12569 catalog_manager.cc:5495] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3adb614ffe84e47aa32619987d9b626 (127.24.186.193). New cstate: current_term: 1 leader_uuid: "b3adb614ffe84e47aa32619987d9b626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3adb614ffe84e47aa32619987d9b626" member_type: VOTER last_known_addr { host: "127.24.186.193" port: 36369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 } health_report { overall_health: UNKNOWN } } }
I20260928 20:28:34.098058 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.095539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.24.186.193:36369)
0928 20:28:34.095604 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:34.095606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:34.095614 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:28:34.095651 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:34.095652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:34.095672 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:34.095737 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:28:34.095906 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.096395 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.097230 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:34.097442 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:34.097505 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:34.097683 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:28:34.097794 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:28:34.097855 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:34.097903 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 20:28:34.097925 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:28:34.100157 12975 raft_consensus.cc:1261] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 1 FOLLOWER]: Refusing update from remote peer b3adb614ffe84e47aa32619987d9b626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:34.100349 12844 raft_consensus.cc:1261] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 1 FOLLOWER]: Refusing update from remote peer b3adb614ffe84e47aa32619987d9b626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:28:34.100589 13045 consensus_queue.cc:1035] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab0d091856b445dd88b712632863d596" member_type: VOTER last_known_addr { host: "127.24.186.195" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:34.100723 13043 consensus_queue.cc:1035] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eec1d9c0d594bf08ec53f8e3f16db05" member_type: VOTER last_known_addr { host: "127.24.186.194" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:28:34.104416 13050 mvcc.cc:205] Tried to move back new op lower bound from 7334409478551343104 to 7334409478483566592. Current Snapshot: MvccSnapshot[applied={T|T < 7334409478551343104 or (T in {7334409478551343104})}]
I20260928 20:28:34.106104 12568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:34.106227 12568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 20:28:34.107085 12568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09e12ef850734c5c83f3dc829d46aea5: Sending DeleteTablet for 3 replicas of tablet e6020337bc954d2e949692abd87efbca
I20260928 20:28:34.107559 12955 tablet_service.cc:1514] Processing DeleteTablet for tablet e6020337bc954d2e949692abd87efbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:34 UTC) from {username='slave'} at 127.0.0.1:41430
I20260928 20:28:34.107586 12692 tablet_service.cc:1514] Processing DeleteTablet for tablet e6020337bc954d2e949692abd87efbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:34 UTC) from {username='slave'} at 127.0.0.1:37924
I20260928 20:28:34.107612 12824 tablet_service.cc:1514] Processing DeleteTablet for tablet e6020337bc954d2e949692abd87efbca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 20:28:34 UTC) from {username='slave'} at 127.0.0.1:52108
I20260928 20:28:34.107908 13059 tablet_replica.cc:330] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626: stopping tablet replica
I20260928 20:28:34.108000 13058 tablet_replica.cc:330] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: stopping tablet replica
I20260928 20:28:34.108032 13059 raft_consensus.cc:2229] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 1 LEADER]: Raft consensus shutting down.
I20260928 20:28:34.108116 13058 raft_consensus.cc:2229] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:34.108179 13059 raft_consensus.cc:2258] T e6020337bc954d2e949692abd87efbca P b3adb614ffe84e47aa32619987d9b626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:34.108186 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12629
I20260928 20:28:34.108235 13058 raft_consensus.cc:2258] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:34.108454 13060 tablet_replica.cc:330] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: stopping tablet replica
I20260928 20:28:34.108556 13060 raft_consensus.cc:2229] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 20:28:34.108644 13060 raft_consensus.cc:2258] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 20:28:34.109099 13060 ts_tablet_manager.cc:1905] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:34.109293 13058 ts_tablet_manager.cc:1905] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 20:28:34.110868 13058 ts_tablet_manager.cc:1918] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:34.110868 13060 ts_tablet_manager.cc:1918] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 20:28:34.110953 13060 log.cc:1196] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-1/wal/wals/e6020337bc954d2e949692abd87efbca
I20260928 20:28:34.111081 13058 log.cc:1196] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Deleting WAL directory at /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790627060930553-25323-0/minicluster-data/ts-2/wal/wals/e6020337bc954d2e949692abd87efbca
I20260928 20:28:34.111218 13060 ts_tablet_manager.cc:1939] T e6020337bc954d2e949692abd87efbca P 4eec1d9c0d594bf08ec53f8e3f16db05: Deleting consensus metadata
I20260928 20:28:34.111444 13058 ts_tablet_manager.cc:1939] T e6020337bc954d2e949692abd87efbca P ab0d091856b445dd88b712632863d596: Deleting consensus metadata
I20260928 20:28:34.111778 12555 catalog_manager.cc:4841] TS 4eec1d9c0d594bf08ec53f8e3f16db05 (127.24.186.194:34255): tablet e6020337bc954d2e949692abd87efbca (table test-table [id=cd6e69ae0e4a436bb09affca1e5e3c66]) successfully deleted
I20260928 20:28:34.111925 12555 catalog_manager.cc:4841] TS ab0d091856b445dd88b712632863d596 (127.24.186.195:45917): tablet e6020337bc954d2e949692abd87efbca (table test-table [id=cd6e69ae0e4a436bb09affca1e5e3c66]) successfully deleted
W20260928 20:28:34.115721 12553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.24.186.193:36369 (error 108)
I20260928 20:28:34.115927 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12761
I20260928 20:28:34.116818 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:34.116674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.193:36369 (local address 127.0.0.1:37944)
0928 20:28:34.116723 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 20:28:34.116774 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.193:36369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 20:28:34.117007 12553 catalog_manager.cc:4584] TS b3adb614ffe84e47aa32619987d9b626 (127.24.186.193:36369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6020337bc954d2e949692abd87efbca: Network error: Client connection negotiation failed: client connection to 127.24.186.193:36369: connect: Connection refused (error 111)
I20260928 20:28:34.120890 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12893
I20260928 20:28:34.126348 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 12537
2026-09-28T20:28:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (817 ms)
[----------] 6 tests from EncryptionPolicyTest (8802 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:34 dist-test-slave-k477 krb5kdc[13064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:34 dist-test-slave-k477 krb5kdc[13064](info): set up 2 sockets
Sep 28 20:28:34 dist-test-slave-k477 krb5kdc[13064](info): commencing operation
krb5kdc: starting...
W20260928 20:28:35.208609 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:35.335209 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:35.425884 13077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:35.426179 13077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:35.426221 13077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:35.427575 13077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:35.427673 13077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:35.427698 13077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:35.427721 13077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:35.427752 13077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:35.429579 13077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:35.429895 13077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:35.430150 13077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:35.435344 13082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:35.435421 13077 server_base.cc:1034] running on GCE node
W20260928 20:28:35.435321 13083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:35.435319 13085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:35.435920 13077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:35.436347 13077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:35.437534 13077 hybrid_clock.cc:648] HybridClock initialized: now 1790627315437518 us; error 54 us; skew 500 ppm
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:35.440562 13077 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:35.441067 13077 webserver.cc:460] Webserver started at http://127.0.0.1:34055/ using document root <none> and password file <none>
I20260928 20:28:35.441308 13077 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:35.441376 13077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:35.441541 13077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:35.442656 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "2003860be11648a4996860e439209941"
format_stamp: "Formatted at 2026-09-28 20:28:35 on dist-test-slave-k477"
server_key: "1e10d4a634e71e6a223c3d26d0a03e45"
server_key_iv: "0be4589f55e2bef89ea959da9394b26e"
server_key_version: "encryptionkey@0"
I20260928 20:28:35.443048 13077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "2003860be11648a4996860e439209941"
format_stamp: "Formatted at 2026-09-28 20:28:35 on dist-test-slave-k477"
server_key: "1e10d4a634e71e6a223c3d26d0a03e45"
server_key_iv: "0be4589f55e2bef89ea959da9394b26e"
server_key_version: "encryptionkey@0"
I20260928 20:28:35.444832 13077 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:28:35.445734 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:35.446002 13077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:35.446141 13077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "2003860be11648a4996860e439209941"
format_stamp: "Formatted at 2026-09-28 20:28:35 on dist-test-slave-k477"
server_key: "1e10d4a634e71e6a223c3d26d0a03e45"
server_key_iv: "0be4589f55e2bef89ea959da9394b26e"
server_key_version: "encryptionkey@0"
I20260928 20:28:35.446228 13077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:35.462953 13077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:35.465452 13077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:35.465633 13077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:35.470577 13077 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38825
I20260928 20:28:35.470703 13144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38825 every 8 connection(s)
I20260928 20:28:35.471032 13077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:35.472196 13145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:35.474592 13145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941: Bootstrap starting.
I20260928 20:28:35.475184 13145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:35.475418 13145 log.cc:824] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:35.476171 13145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941: No bootstrap required, opened a new log
I20260928 20:28:35.478493 13145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } }
I20260928 20:28:35.478657 13145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:35.478688 13145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2003860be11648a4996860e439209941, State: Initialized, Role: FOLLOWER
I20260928 20:28:35.478803 13145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } }
I20260928 20:28:35.478904 13145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:35.478941 13145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:35.478999 13145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:35.479745 13145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } }
I20260928 20:28:35.479936 13145 leader_election.cc:302] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2003860be11648a4996860e439209941; no voters:
I20260928 20:28:35.480149 13145 leader_election.cc:288] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:35.480304 13148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:35.480453 13145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:35.480680 13148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [term 1 LEADER]: Becoming Leader. State: Replica: 2003860be11648a4996860e439209941, State: Running, Role: LEADER
I20260928 20:28:35.480799 13148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } }
I20260928 20:28:35.480921 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13077
I20260928 20:28:35.481072 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:35.481302 25323 external_mini_cluster.cc:1428] Setting key 343afe8c1ecd34400816170cfa8a146f
I20260928 20:28:35.481278 13150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2003860be11648a4996860e439209941. Latest consensus state: current_term: 1 leader_uuid: "2003860be11648a4996860e439209941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } } }
I20260928 20:28:35.481371 13150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:35.481556 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2003860be11648a4996860e439209941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2003860be11648a4996860e439209941" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38825 } } }
I20260928 20:28:35.481693 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:35.482161 13152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:35.482750 13152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:35.484473 13152 catalog_manager.cc:1347] Generated new cluster ID: 0946189b3b8949dba649e75a331fbd54
I20260928 20:28:35.484529 13152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:28:35.495714 13152 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:35.496258 13152 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:35.496464 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:35.482059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59187 (local address 127.0.0.1:38825)
0928 20:28:35.482244 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:35.482255 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:35.482287 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:28:35.482422 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:35.482430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:35.482505 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:35.483465 (+ 960us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:35.483542 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:35.484822 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:35.486181 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:35.488620 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:35.488752 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:35.491860 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:35.491868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:35.491868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:35.491888 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:35.494016 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:35.494494 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:35.494498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:35.494499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:35.494627 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:35.494948 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:35.494949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:35.494950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:35.495300 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:35.495982 (+ 682us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:35.496074 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 20:28:35.496188 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 20:28:35.497532 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:35.501358 13152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2003860be11648a4996860e439209941: Generated new TSK 0
I20260928 20:28:35.501590 13152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:35.507232 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:35.499105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46419 (local address 127.0.0.1:38825)
0928 20:28:35.499146 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:35.499148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:35.499288 (+ 140us) server_negotiation.cc:407] Connection header received
0928 20:28:35.499317 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:35.499318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:35.499328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:35.499400 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:35.500022 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:35.500865 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:35.501990 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:35.502202 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:35.502447 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:35.504002 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:35.504003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:35.504004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:35.504011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:35.505544 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:35.506429 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:35.506430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:35.506431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:35.506493 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:35.506764 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:35.506765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:35.506765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:35.506873 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:35.506952 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:35.507112 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 20:28:35.507164 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:28:35.510143 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13077
W20260928 20:28:35.515214 13062 connection.cc:535] client connection to 127.0.0.1:38825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38825): Connection reset by peer (error 104)
2026-09-28T20:28:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13176](info): set up 2 sockets
Sep 28 20:28:35 dist-test-slave-k477 krb5kdc[13176](info): commencing operation
krb5kdc: starting...
W20260928 20:28:36.572530 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:36 dist-test-slave-k477 krb5kdc[13176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:36.700519 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:36.784106 13189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:36.784381 13189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:36.784423 13189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:36.785629 13189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:36.785712 13189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:36.785738 13189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:36.785770 13189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:36.785795 13189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:36.787367 13189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:36.787679 13189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:36.787983 13189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:36.793244 13197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:36.793265 13194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:36.793452 13195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:36.793589 13189 server_base.cc:1034] running on GCE node
I20260928 20:28:36.793851 13189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:36.794157 13189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:36.795348 13189 hybrid_clock.cc:648] HybridClock initialized: now 1790627316795316 us; error 50 us; skew 500 ppm
Sep 28 20:28:36 dist-test-slave-k477 krb5kdc[13176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:36.798760 13189 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:36.799283 13189 webserver.cc:460] Webserver started at http://127.0.0.1:45485/ using document root <none> and password file <none>
I20260928 20:28:36.799520 13189 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:36.799590 13189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:36.799752 13189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:36.800885 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "81ea99cae55e438ab8fe67f07692d53c"
format_stamp: "Formatted at 2026-09-28 20:28:36 on dist-test-slave-k477"
server_key: "668349e726ba4268403d79c441a64ec0"
server_key_iv: "c1dac2b32ed90ebbd9624cf823aca069"
server_key_version: "encryptionkey@0"
I20260928 20:28:36.801270 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "81ea99cae55e438ab8fe67f07692d53c"
format_stamp: "Formatted at 2026-09-28 20:28:36 on dist-test-slave-k477"
server_key: "668349e726ba4268403d79c441a64ec0"
server_key_iv: "c1dac2b32ed90ebbd9624cf823aca069"
server_key_version: "encryptionkey@0"
I20260928 20:28:36.802723 13189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:36.803635 13204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:36.803891 13189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:36.804001 13189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "81ea99cae55e438ab8fe67f07692d53c"
format_stamp: "Formatted at 2026-09-28 20:28:36 on dist-test-slave-k477"
server_key: "668349e726ba4268403d79c441a64ec0"
server_key_iv: "c1dac2b32ed90ebbd9624cf823aca069"
server_key_version: "encryptionkey@0"
I20260928 20:28:36.804096 13189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:36.828739 13189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:36.831445 13189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:36.831643 13189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:36.835866 13189 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39885
I20260928 20:28:36.835889 13256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39885 every 8 connection(s)
I20260928 20:28:36.836288 13189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:36.836627 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13189
I20260928 20:28:36.836793 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:36.836957 25323 external_mini_cluster.cc:1428] Setting key 4ca963cd0c9068426a1753ee6b8c64ea
I20260928 20:28:36.837551 13257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:36.840664 13257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c: Bootstrap starting.
I20260928 20:28:36.841596 13257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:36.841964 13257 log.cc:824] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:36.842610 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:36.837446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38653 (local address 127.0.0.1:39885)
0928 20:28:36.837636 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:36.837652 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:36.837678 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:28:36.838109 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:36.838115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:36.838160 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:36.838306 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:36.838522 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:36.839449 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:36.840303 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:36.840714 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:36.840835 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:36.841895 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:36.841899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:36.841899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:36.842124 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:36.842156 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:36.842169 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:36.842263 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:36.842399 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 20:28:36.842428 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260928 20:28:36.843611 13257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c: No bootstrap required, opened a new log
I20260928 20:28:36.843653 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:36.846012 13257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260928 20:28:36.846228 13257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:36.846271 13257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81ea99cae55e438ab8fe67f07692d53c, State: Initialized, Role: FOLLOWER
I20260928 20:28:36.846397 13257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260928 20:28:36.846481 13257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:36.846514 13257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:36.846575 13257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:36.847427 13257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260928 20:28:36.847568 13257 leader_election.cc:302] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81ea99cae55e438ab8fe67f07692d53c; no voters:
I20260928 20:28:36.847862 13257 leader_election.cc:288] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:36.847901 13267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:36.848078 13267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [term 1 LEADER]: Becoming Leader. State: Replica: 81ea99cae55e438ab8fe67f07692d53c, State: Running, Role: LEADER
I20260928 20:28:36.848142 13257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:36.848313 13267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } }
I20260928 20:28:36.848598 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:36.845215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36501 (local address 127.0.0.1:39885)
0928 20:28:36.845337 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:36.845339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:36.845354 (+ 15us) server_negotiation.cc:407] Connection header received
0928 20:28:36.845379 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:36.845380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:36.845391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:36.845453 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:36.845599 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:36.846080 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:36.847078 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:36.847271 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:36.847326 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:36.848164 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:36.848166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:36.848166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:36.848172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:36.848187 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:36.848192 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:36.848241 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:36.848495 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 20:28:36.848511 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 20:28:36.848897 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81ea99cae55e438ab8fe67f07692d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260928 20:28:36.848904 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81ea99cae55e438ab8fe67f07692d53c. Latest consensus state: current_term: 1 leader_uuid: "81ea99cae55e438ab8fe67f07692d53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81ea99cae55e438ab8fe67f07692d53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39885 } } }
I20260928 20:28:36.848995 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:36.849009 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:36.849273 13273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:36.849792 13273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:36.851414 13273 catalog_manager.cc:1347] Generated new cluster ID: d78e1fb76c444cf5bbe65cfe59ccc629
I20260928 20:28:36.851465 13273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:36.855909 13273 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:36.856734 13273 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:36.865775 13273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81ea99cae55e438ab8fe67f07692d53c: Generated new TSK 0
I20260928 20:28:36.865993 13273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:36.874902 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13189
W20260928 20:28:36.879240 13174 connection.cc:535] client connection to 127.0.0.1:39885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39885): Connection reset by peer (error 104)
2026-09-28T20:28:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:36 dist-test-slave-k477 krb5kdc[13288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:36 dist-test-slave-k477 krb5kdc[13288](info): set up 2 sockets
Sep 28 20:28:36 dist-test-slave-k477 krb5kdc[13288](info): commencing operation
krb5kdc: starting...
W20260928 20:28:37.964656 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:38 dist-test-slave-k477 krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:38.092211 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:38.181339 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:38.181591 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:38.181617 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:38.182874 13301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:38.182929 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:38.182943 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:38.182956 13301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:38.182968 13301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:38.184587 13301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:38.184877 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:38.185118 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:38.190780 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:38.190784 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:38.190907 13301 server_base.cc:1034] running on GCE node
W20260928 20:28:38.190754 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:38.191270 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:38.191579 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:38.192749 13301 hybrid_clock.cc:648] HybridClock initialized: now 1790627318192713 us; error 57 us; skew 500 ppm
Sep 28 20:28:38 dist-test-slave-k477 krb5kdc[13288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:38.195681 13301 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:38.196182 13301 webserver.cc:460] Webserver started at http://127.0.0.1:43457/ using document root <none> and password file <none>
I20260928 20:28:38.196434 13301 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:38.196480 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:38.196662 13301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:38.197764 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd"
format_stamp: "Formatted at 2026-09-28 20:28:38 on dist-test-slave-k477"
server_key: "363d3a142a98168a5d2f28c75f748b54"
server_key_iv: "558a4c7cf656ddb86b29d619bda57471"
server_key_version: "encryptionkey@0"
I20260928 20:28:38.198127 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd"
format_stamp: "Formatted at 2026-09-28 20:28:38 on dist-test-slave-k477"
server_key: "363d3a142a98168a5d2f28c75f748b54"
server_key_iv: "558a4c7cf656ddb86b29d619bda57471"
server_key_version: "encryptionkey@0"
I20260928 20:28:38.199505 13301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:38.200639 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:38.200938 13301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:38.201068 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd"
format_stamp: "Formatted at 2026-09-28 20:28:38 on dist-test-slave-k477"
server_key: "363d3a142a98168a5d2f28c75f748b54"
server_key_iv: "558a4c7cf656ddb86b29d619bda57471"
server_key_version: "encryptionkey@0"
I20260928 20:28:38.201151 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:38.204855 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:38.206981 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:38.207154 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:38.212250 13301 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41575
I20260928 20:28:38.212316 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41575 every 8 connection(s)
I20260928 20:28:38.212677 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:38.213585 13369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:38.216372 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd: Bootstrap starting.
I20260928 20:28:38.217025 13369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:38.217321 13369 log.cc:824] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:38.217639 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13301
I20260928 20:28:38.217799 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:38.218005 25323 external_mini_cluster.cc:1428] Setting key 1c17103e00b23ca0770502ed755ea17e
I20260928 20:28:38.218118 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd: No bootstrap required, opened a new log
I20260928 20:28:38.220654 13369 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } }
I20260928 20:28:38.220844 13369 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:38.220887 13369 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43be9b4922d4a3eb80c0298ab6bd8bd, State: Initialized, Role: FOLLOWER
I20260928 20:28:38.221230 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:38.218602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35873 (local address 127.0.0.1:41575)
0928 20:28:38.218815 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:38.218826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:38.218865 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:28:38.219598 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:38.219607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:38.219642 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:38.219828 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:38.220702 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:38.220706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:38.220706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:38.220710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:38.220754 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:38.220767 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:38.220852 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:38.220959 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 20:28:38.221003 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 20:28:38.221071 13369 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } }
I20260928 20:28:38.221503 13369 raft_consensus.cc:388] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:38.221587 13369 raft_consensus.cc:482] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:38.221639 13369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:38.222203 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:38.222442 13369 raft_consensus.cc:504] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } }
I20260928 20:28:38.222571 13369 leader_election.cc:302] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b43be9b4922d4a3eb80c0298ab6bd8bd; no voters:
I20260928 20:28:38.222775 13369 leader_election.cc:288] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:38.222796 13374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:38.222908 13374 raft_consensus.cc:686] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [term 1 LEADER]: Becoming Leader. State: Replica: b43be9b4922d4a3eb80c0298ab6bd8bd, State: Running, Role: LEADER
I20260928 20:28:38.223038 13369 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:38.223037 13374 consensus_queue.cc:237] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } }
I20260928 20:28:38.223383 13375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } } }
I20260928 20:28:38.223389 13376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b43be9b4922d4a3eb80c0298ab6bd8bd. Latest consensus state: current_term: 1 leader_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43be9b4922d4a3eb80c0298ab6bd8bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41575 } } }
I20260928 20:28:38.223542 13375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:38.223564 13376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:38.223912 13382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:38.224447 13382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:38.225253 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:38.223837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35899 (local address 127.0.0.1:41575)
0928 20:28:38.223872 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:38.223874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:38.224109 (+ 235us) server_negotiation.cc:407] Connection header received
0928 20:28:38.224214 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:38.224214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:38.224220 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:38.224264 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:38.224985 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:38.224986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:38.224986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:38.224990 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:38.225007 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:38.225011 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:38.225071 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:38.225164 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 20:28:38.225175 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:28:38.226188 13382 catalog_manager.cc:1347] Generated new cluster ID: 3201c7ef9ee143a68b2dc8914bcc24b8
I20260928 20:28:38.226238 13382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:38.233968 13382 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:38.234177 13382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:38.238866 13382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b43be9b4922d4a3eb80c0298ab6bd8bd: Generated new TSK 0
I20260928 20:28:38.239078 13382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:38.243764 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13301
2026-09-28T20:28:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:38 dist-test-slave-k477 krb5kdc[13400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:38 dist-test-slave-k477 krb5kdc[13400](info): set up 2 sockets
Sep 28 20:28:38 dist-test-slave-k477 krb5kdc[13400](info): commencing operation
krb5kdc: starting...
W20260928 20:28:39.308565 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:39.441670 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:39.530243 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:39.530489 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:39.530516 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:39.531723 13413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:39.531771 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:39.531785 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:39.531867 13413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:39.531894 13413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:39.533505 13413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:39.533800 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:39.534077 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:39.540077 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:39.540073 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:39.540067 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:39.540342 13413 server_base.cc:1034] running on GCE node
I20260928 20:28:39.540591 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:39.540966 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:39.542167 13413 hybrid_clock.cc:648] HybridClock initialized: now 1790627319542148 us; error 47 us; skew 500 ppm
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:39.545989 13413 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:39.546573 13413 webserver.cc:460] Webserver started at http://127.0.0.1:42821/ using document root <none> and password file <none>
I20260928 20:28:39.546818 13413 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:39.546865 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:39.546964 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:39.548179 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "974bbb56648c4b3693bbf9df09051f51"
format_stamp: "Formatted at 2026-09-28 20:28:39 on dist-test-slave-k477"
server_key: "dc4c66f8fe682e3ac34318546ffa982f"
server_key_iv: "9e16ebd356fd1cbd8208cd42389255cf"
server_key_version: "encryptionkey@0"
I20260928 20:28:39.548481 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "974bbb56648c4b3693bbf9df09051f51"
format_stamp: "Formatted at 2026-09-28 20:28:39 on dist-test-slave-k477"
server_key: "dc4c66f8fe682e3ac34318546ffa982f"
server_key_iv: "9e16ebd356fd1cbd8208cd42389255cf"
server_key_version: "encryptionkey@0"
I20260928 20:28:39.549804 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:39.550779 13428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:39.551208 13413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:39.551335 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "974bbb56648c4b3693bbf9df09051f51"
format_stamp: "Formatted at 2026-09-28 20:28:39 on dist-test-slave-k477"
server_key: "dc4c66f8fe682e3ac34318546ffa982f"
server_key_iv: "9e16ebd356fd1cbd8208cd42389255cf"
server_key_version: "encryptionkey@0"
I20260928 20:28:39.551394 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:39.561579 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:39.564898 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:39.565060 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:39.569514 13413 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42219
I20260928 20:28:39.569571 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42219 every 8 connection(s)
I20260928 20:28:39.569965 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:39.570863 13481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:39.573367 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51: Bootstrap starting.
I20260928 20:28:39.573993 13481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:39.574290 13481 log.cc:824] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:39.575127 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51: No bootstrap required, opened a new log
I20260928 20:28:39.576934 13481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } }
I20260928 20:28:39.577082 13481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:39.577105 13481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974bbb56648c4b3693bbf9df09051f51, State: Initialized, Role: FOLLOWER
I20260928 20:28:39.577275 13481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } }
I20260928 20:28:39.577359 13481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:39.577406 13481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:39.577458 13481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:39.577723 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13413
I20260928 20:28:39.577864 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:39.578047 25323 external_mini_cluster.cc:1428] Setting key f6664cd2d4420410e969327e45d0b205
I20260928 20:28:39.578187 13481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } }
I20260928 20:28:39.578317 13481 leader_election.cc:302] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 974bbb56648c4b3693bbf9df09051f51; no voters:
I20260928 20:28:39.578516 13481 leader_election.cc:288] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:39.578697 13486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:39.578804 13481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:39.578936 13486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [term 1 LEADER]: Becoming Leader. State: Replica: 974bbb56648c4b3693bbf9df09051f51, State: Running, Role: LEADER
I20260928 20:28:39.579084 13486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } }
I20260928 20:28:39.579483 13488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "974bbb56648c4b3693bbf9df09051f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } } }
I20260928 20:28:39.579569 13488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:39.579730 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 974bbb56648c4b3693bbf9df09051f51. Latest consensus state: current_term: 1 leader_uuid: "974bbb56648c4b3693bbf9df09051f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "974bbb56648c4b3693bbf9df09051f51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42219 } } }
I20260928 20:28:39.579829 13491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:39.579923 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:39.580504 13491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:39.582247 13491 catalog_manager.cc:1347] Generated new cluster ID: e57179a10d77440ebcd185bacf8435c4
I20260928 20:28:39.582301 13491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:28:39.589589 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:39.578755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52629 (local address 127.0.0.1:42219)
0928 20:28:39.578915 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:39.578925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:39.578959 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:28:39.579100 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:39.579107 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:39.579182 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:39.579417 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:39.580287 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:39.581831 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:39.582815 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:39.583262 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:39.583380 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:39.585977 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:39.585982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:39.585983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:39.586005 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:39.587941 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:39.588344 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:39.588348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:39.588349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:39.588459 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:39.588698 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:39.588699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:39.588700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:39.588972 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:39.589157 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:39.589237 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 20:28:39.589391 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 20:28:39.590538 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:39.594457 13491 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:39.595238 13491 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:39.605111 13491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 974bbb56648c4b3693bbf9df09051f51: Generated new TSK 0
I20260928 20:28:39.605361 13491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:39.607627 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:39.592033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59311 (local address 127.0.0.1:42219)
0928 20:28:39.592074 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:39.592076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:39.600125 (+ 8049us) server_negotiation.cc:407] Connection header received
0928 20:28:39.600222 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:39.600223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:39.600380 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:39.600564 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:39.601378 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:39.602070 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:39.603001 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:39.603247 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:39.603328 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:39.604703 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:39.604705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:39.604705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:39.604714 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:39.606420 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:39.606807 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:39.606808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:39.606808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:39.606871 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:39.607054 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:39.607055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:39.607055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:39.607200 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:39.607316 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:39.607447 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:28:39.607532 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:28:39.609705 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13413
2026-09-28T20:28:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13512](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13512](info): set up 2 sockets
Sep 28 20:28:39 dist-test-slave-k477 krb5kdc[13512](info): commencing operation
krb5kdc: starting...
W20260928 20:28:40.684672 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:40 dist-test-slave-k477 krb5kdc[13512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:40.815864 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:40.903053 13525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:40.903301 13525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:40.903327 13525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:40.904855 13525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:40.904925 13525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:40.904942 13525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:40.904956 13525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:40.904969 13525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:40.906659 13525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:40.906890 13525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:40.907074 13525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:40.912168 13531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:40.912371 13530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:40.912299 13533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:40.912479 13525 server_base.cc:1034] running on GCE node
I20260928 20:28:40.913060 13525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:40.913429 13525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:40.914623 13525 hybrid_clock.cc:648] HybridClock initialized: now 1790627320914599 us; error 41 us; skew 500 ppm
Sep 28 20:28:40 dist-test-slave-k477 krb5kdc[13512](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:40.924551 13525 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:40.925047 13525 webserver.cc:460] Webserver started at http://127.0.0.1:37027/ using document root <none> and password file <none>
I20260928 20:28:40.925304 13525 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:40.925354 13525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:40.925482 13525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:40.926674 13525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "752aa37cccc640b7ba64ebdce6ac8d5f"
format_stamp: "Formatted at 2026-09-28 20:28:40 on dist-test-slave-k477"
server_key: "a55323f3c0d1a30ade7f7430a87adbc7"
server_key_iv: "2b66bcff8133aac46c06fa156ed56235"
server_key_version: "encryptionkey@0"
I20260928 20:28:40.927083 13525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "752aa37cccc640b7ba64ebdce6ac8d5f"
format_stamp: "Formatted at 2026-09-28 20:28:40 on dist-test-slave-k477"
server_key: "a55323f3c0d1a30ade7f7430a87adbc7"
server_key_iv: "2b66bcff8133aac46c06fa156ed56235"
server_key_version: "encryptionkey@0"
I20260928 20:28:40.928655 13525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:40.929509 13540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:40.929724 13525 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:40.929842 13525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "752aa37cccc640b7ba64ebdce6ac8d5f"
format_stamp: "Formatted at 2026-09-28 20:28:40 on dist-test-slave-k477"
server_key: "a55323f3c0d1a30ade7f7430a87adbc7"
server_key_iv: "2b66bcff8133aac46c06fa156ed56235"
server_key_version: "encryptionkey@0"
I20260928 20:28:40.929935 13525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:40.942091 13525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:40.945897 13525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:40.946115 13525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:40.952492 13525 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41527
I20260928 20:28:40.952562 13592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41527 every 8 connection(s)
I20260928 20:28:40.952997 13525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:40.954054 13593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:40.956534 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f: Bootstrap starting.
I20260928 20:28:40.957166 13593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:40.957458 13593 log.cc:824] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:40.958195 13593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f: No bootstrap required, opened a new log
I20260928 20:28:40.959860 13593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260928 20:28:40.960021 13593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:40.960044 13593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752aa37cccc640b7ba64ebdce6ac8d5f, State: Initialized, Role: FOLLOWER
I20260928 20:28:40.960224 13593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260928 20:28:40.960287 13593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:40.960309 13593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:40.960379 13593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:40.960986 13593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260928 20:28:40.961074 13593 leader_election.cc:302] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 752aa37cccc640b7ba64ebdce6ac8d5f; no voters:
I20260928 20:28:40.961305 13593 leader_election.cc:288] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:40.961459 13596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:40.961606 13593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:40.961884 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13525
I20260928 20:28:40.961648 13596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [term 1 LEADER]: Becoming Leader. State: Replica: 752aa37cccc640b7ba64ebdce6ac8d5f, State: Running, Role: LEADER
I20260928 20:28:40.962031 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:40.962046 13596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } }
I20260928 20:28:40.962285 25323 external_mini_cluster.cc:1428] Setting key 8f7909d9eafb8920f4555e1a8250f1ed
I20260928 20:28:40.962441 13598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752aa37cccc640b7ba64ebdce6ac8d5f. Latest consensus state: current_term: 1 leader_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } } }
I20260928 20:28:40.962524 13598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:40.962769 13600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:40.963121 13597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752aa37cccc640b7ba64ebdce6ac8d5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41527 } } }
I20260928 20:28:40.963215 13597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:40.963339 13600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:40.965106 13600 catalog_manager.cc:1347] Generated new cluster ID: ab838671cf0b46a293f65d410ebfb366
I20260928 20:28:40.965165 13600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:40.968984 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:40.962833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43717 (local address 127.0.0.1:41527)
0928 20:28:40.964367 (+ 1534us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:40.964378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:40.964415 (+ 37us) server_negotiation.cc:407] Connection header received
0928 20:28:40.964540 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:40.964548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:40.964612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:40.964804 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:40.964992 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:40.965971 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:40.966803 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:40.967341 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:40.967500 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:40.968297 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:40.968301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:40.968302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:40.968490 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:40.968522 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:40.968534 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:40.968614 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:40.968689 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:28:40.968758 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1483,"thread_start_us":78,"threads_started":1}
I20260928 20:28:40.969916 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:40.974720 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:40.971329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53091 (local address 127.0.0.1:41527)
0928 20:28:40.971367 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:40.971369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:40.971476 (+ 107us) server_negotiation.cc:407] Connection header received
0928 20:28:40.971534 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:40.971535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:40.971546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:40.971590 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:40.971748 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:40.972298 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:40.973072 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:40.973291 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:40.973369 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:40.974130 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:40.974131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:40.974132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:40.974136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:40.974151 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:40.974155 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:40.974226 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:40.974541 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 20:28:40.974592 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:28:40.984225 13600 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:40.984969 13600 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:40.991688 13600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 752aa37cccc640b7ba64ebdce6ac8d5f: Generated new TSK 0
I20260928 20:28:40.991932 13600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:40.995775 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13525
2026-09-28T20:28:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:41 dist-test-slave-k477 krb5kdc[13624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:41 dist-test-slave-k477 krb5kdc[13624](info): set up 2 sockets
Sep 28 20:28:41 dist-test-slave-k477 krb5kdc[13624](info): commencing operation
krb5kdc: starting...
W20260928 20:28:42.064908 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:42 dist-test-slave-k477 krb5kdc[13624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:42.198441 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39793
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:42.284229 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:42.284498 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:42.284549 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:42.285853 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:42.285933 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:42.285959 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:42.285988 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:42.286017 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:42.287614 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:42.287941 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:42.288190 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:42.293323 13643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:42.293339 13645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:42.293320 13642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:42.293596 13637 server_base.cc:1034] running on GCE node
I20260928 20:28:42.293898 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:42.294378 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:42.295595 13637 hybrid_clock.cc:648] HybridClock initialized: now 1790627322295579 us; error 29 us; skew 500 ppm
Sep 28 20:28:42 dist-test-slave-k477 krb5kdc[13624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:42.298959 13637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:42.299434 13637 webserver.cc:460] Webserver started at http://127.0.0.1:33021/ using document root <none> and password file <none>
I20260928 20:28:42.299700 13637 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:42.299772 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:42.299983 13637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:42.301121 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "f38936ddfa07408788d43c1f241f4513"
format_stamp: "Formatted at 2026-09-28 20:28:42 on dist-test-slave-k477"
server_key: "1deab937f2e89f99f39385d93a0fb11f"
server_key_iv: "6b739618580e6378e7027015b93faba1"
server_key_version: "encryptionkey@0"
I20260928 20:28:42.301491 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "f38936ddfa07408788d43c1f241f4513"
format_stamp: "Formatted at 2026-09-28 20:28:42 on dist-test-slave-k477"
server_key: "1deab937f2e89f99f39385d93a0fb11f"
server_key_iv: "6b739618580e6378e7027015b93faba1"
server_key_version: "encryptionkey@0"
I20260928 20:28:42.302914 13637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:42.303835 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:42.304034 13637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:42.304140 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "f38936ddfa07408788d43c1f241f4513"
format_stamp: "Formatted at 2026-09-28 20:28:42 on dist-test-slave-k477"
server_key: "1deab937f2e89f99f39385d93a0fb11f"
server_key_iv: "6b739618580e6378e7027015b93faba1"
server_key_version: "encryptionkey@0"
I20260928 20:28:42.304229 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:42.307550 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:42.309911 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:42.310077 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:42.314172 13637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39793
I20260928 20:28:42.314204 13704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39793 every 8 connection(s)
I20260928 20:28:42.314548 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:42.315531 13705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:42.317922 13705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513: Bootstrap starting.
I20260928 20:28:42.318564 13705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:42.318868 13705 log.cc:824] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:42.319761 13705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513: No bootstrap required, opened a new log
I20260928 20:28:42.321453 13705 raft_consensus.cc:348] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } }
I20260928 20:28:42.321607 13705 raft_consensus.cc:374] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:42.321631 13705 raft_consensus.cc:729] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38936ddfa07408788d43c1f241f4513, State: Initialized, Role: FOLLOWER
I20260928 20:28:42.321820 13705 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } }
I20260928 20:28:42.321899 13705 raft_consensus.cc:388] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:42.321949 13705 raft_consensus.cc:482] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:42.322000 13705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:42.322573 13705 raft_consensus.cc:504] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } }
I20260928 20:28:42.322683 13705 leader_election.cc:302] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38936ddfa07408788d43c1f241f4513; no voters:
I20260928 20:28:42.322870 13705 leader_election.cc:288] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:42.322917 13708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:42.323135 13708 raft_consensus.cc:686] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [term 1 LEADER]: Becoming Leader. State: Replica: f38936ddfa07408788d43c1f241f4513, State: Running, Role: LEADER
I20260928 20:28:42.323134 13705 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:42.323256 13708 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } }
I20260928 20:28:42.323648 13710 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38936ddfa07408788d43c1f241f4513. Latest consensus state: current_term: 1 leader_uuid: "f38936ddfa07408788d43c1f241f4513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } } }
I20260928 20:28:42.323899 13710 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:42.323964 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13637
I20260928 20:28:42.323647 13709 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f38936ddfa07408788d43c1f241f4513" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38936ddfa07408788d43c1f241f4513" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39793 } } }
I20260928 20:28:42.324069 13709 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:42.324101 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:42.324185 13713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:42.324328 25323 external_mini_cluster.cc:1428] Setting key 37c0931dd8c2b5b3d9b9aff310259b35
I20260928 20:28:42.324748 13713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:42.326689 13713 catalog_manager.cc:1347] Generated new cluster ID: e85688a895424550818a5d54cdc41de6
I20260928 20:28:42.326761 13713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:42.327297 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:42.325014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45191 (local address 127.0.0.1:39793)
0928 20:28:42.325803 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:42.325812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:42.325843 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:28:42.325955 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:42.325962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:42.325992 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:42.326131 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:42.326886 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:42.326889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:42.326890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:42.326894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:42.326936 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:42.326947 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:42.327022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:42.327097 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:28:42.327140 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":62,"threads_started":1}
I20260928 20:28:42.328130 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:42.337543 13727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:42.336276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47211 (local address 127.0.0.1:39793)
0928 20:28:42.336337 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:42.336340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:42.336480 (+ 140us) server_negotiation.cc:407] Connection header received
0928 20:28:42.336555 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:42.336556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:42.336569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:42.336637 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:42.337288 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:42.337289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:42.337289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:42.337295 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:42.337310 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:42.337315 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:42.337372 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:42.337429 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 20:28:42.337442 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 20:28:42.340016 13713 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:42.340195 13713 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:42.347363 13713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f38936ddfa07408788d43c1f241f4513: Generated new TSK 0
I20260928 20:28:42.347576 13713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:42.354629 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13637
2026-09-28T20:28:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:42 dist-test-slave-k477 krb5kdc[13736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:42 dist-test-slave-k477 krb5kdc[13736](info): set up 2 sockets
Sep 28 20:28:42 dist-test-slave-k477 krb5kdc[13736](info): commencing operation
krb5kdc: starting...
W20260928 20:28:43.424685 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:43.560361 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:43.655835 13749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:43.656174 13749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:43.656219 13749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:43.657704 13749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:43.657769 13749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:43.657784 13749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:43.657799 13749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:43.657811 13749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:43.659484 13749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:43.659833 13749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:43.660099 13749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:43.666440 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:43.666450 13757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:43.666450 13754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:43.666939 13749 server_base.cc:1034] running on GCE node
I20260928 20:28:43.667192 13749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:43.667522 13749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:43.668697 13749 hybrid_clock.cc:648] HybridClock initialized: now 1790627323668705 us; error 54 us; skew 500 ppm
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:43.671285 13749 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:43.671746 13749 webserver.cc:460] Webserver started at http://127.0.0.1:33457/ using document root <none> and password file <none>
I20260928 20:28:43.672024 13749 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:43.672096 13749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:43.672260 13749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:43.673389 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "c4565500ec1047a0aea205aa944d6384"
format_stamp: "Formatted at 2026-09-28 20:28:43 on dist-test-slave-k477"
server_key: "8da7f74fe0069e8f992df47d86ed3503"
server_key_iv: "a6b2e890042575b3b9818fed19be6efa"
server_key_version: "encryptionkey@0"
I20260928 20:28:43.673787 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "c4565500ec1047a0aea205aa944d6384"
format_stamp: "Formatted at 2026-09-28 20:28:43 on dist-test-slave-k477"
server_key: "8da7f74fe0069e8f992df47d86ed3503"
server_key_iv: "a6b2e890042575b3b9818fed19be6efa"
server_key_version: "encryptionkey@0"
I20260928 20:28:43.675269 13749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:43.676178 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:43.676407 13749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:43.676520 13749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "c4565500ec1047a0aea205aa944d6384"
format_stamp: "Formatted at 2026-09-28 20:28:43 on dist-test-slave-k477"
server_key: "8da7f74fe0069e8f992df47d86ed3503"
server_key_iv: "a6b2e890042575b3b9818fed19be6efa"
server_key_version: "encryptionkey@0"
I20260928 20:28:43.676599 13749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:43.695592 13749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:43.697898 13749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:43.698083 13749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:43.702919 13749 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43493
I20260928 20:28:43.702953 13816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43493 every 8 connection(s)
I20260928 20:28:43.703334 13749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:43.704346 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:43.706650 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: Bootstrap starting.
I20260928 20:28:43.706943 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13749
I20260928 20:28:43.707085 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:43.707274 25323 external_mini_cluster.cc:1428] Setting key a78ddd65ca2cb4a5b307de57acc71f29
I20260928 20:28:43.707430 13817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:43.707753 13817 log.cc:824] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:43.708488 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: No bootstrap required, opened a new log
I20260928 20:28:43.710337 13817 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260928 20:28:43.710508 13817 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:43.710559 13817 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4565500ec1047a0aea205aa944d6384, State: Initialized, Role: FOLLOWER
I20260928 20:28:43.710717 13817 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260928 20:28:43.710812 13817 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:43.710857 13817 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:43.710907 13817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:43.711485 13817 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260928 20:28:43.711611 13817 leader_election.cc:302] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4565500ec1047a0aea205aa944d6384; no voters:
I20260928 20:28:43.711850 13817 leader_election.cc:288] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:43.711869 13822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:43.711987 13822 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [term 1 LEADER]: Becoming Leader. State: Replica: c4565500ec1047a0aea205aa944d6384, State: Running, Role: LEADER
I20260928 20:28:43.712133 13817 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:43.712128 13822 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } }
I20260928 20:28:43.714133 13823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4565500ec1047a0aea205aa944d6384. Latest consensus state: current_term: 1 leader_uuid: "c4565500ec1047a0aea205aa944d6384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260928 20:28:43.714393 13823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [sys.catalog]: This master's current role is: LEADER
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260928 20:28:43.714629 13836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:28:43.714716 13836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:43.714695 13824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4565500ec1047a0aea205aa944d6384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4565500ec1047a0aea205aa944d6384" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43493 } } }
I20260928 20:28:43.714879 13839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:43.714950 13824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:43.715193 13839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:43.716949 13839 catalog_manager.cc:1347] Generated new cluster ID: d17659b464c0472fa6a5ee4df1526d3b
I20260928 20:28:43.717005 13839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:43.718892 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:43.707869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:57119 (local address 127.0.0.1:43493)
0928 20:28:43.708081 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:43.708090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:43.708124 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:28:43.708601 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:43.708609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:43.708678 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:43.708820 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:43.709469 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:43.710386 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:43.711306 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:43.711879 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:43.711987 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:43.715229 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:43.715234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:43.715234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:43.715248 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:43.717277 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:43.717689 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:43.717692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:43.717692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:43.717791 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:43.718027 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:43.718028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:43.718028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:43.718273 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:43.718465 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:43.718575 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 20:28:43.718681 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260928 20:28:43.720037 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:43.727314 13839 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:43.728071 13839 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:43.730768 13820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:43.724191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:47173 (local address 127.0.0.1:43493)
0928 20:28:43.724255 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:43.724257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:43.724387 (+ 130us) server_negotiation.cc:407] Connection header received
0928 20:28:43.724458 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:43.724459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:43.724470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:43.724525 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:43.725095 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:43.725602 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:43.726311 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:43.726543 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:43.726627 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:43.728098 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:43.728100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:43.728100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:43.728107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:43.729569 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:43.729940 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:43.729941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:43.729941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:43.730004 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:43.730236 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:43.730238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:43.730238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:43.730354 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:43.730467 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:43.730598 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:28:43.730662 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:28:43.736358 13839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4565500ec1047a0aea205aa944d6384: Generated new TSK 0
I20260928 20:28:43.736579 13839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:43.740404 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13749
W20260928 20:28:43.744999 13734 connection.cc:535] client connection to 127.0.0.1:43493 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43493): Connection reset by peer (error 104)
2026-09-28T20:28:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13848](info): set up 2 sockets
Sep 28 20:28:43 dist-test-slave-k477 krb5kdc[13848](info): commencing operation
krb5kdc: starting...
W20260928 20:28:44.796578 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:44 dist-test-slave-k477 krb5kdc[13848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:44.924764 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:45.014413 13861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:45.014678 13861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:45.014726 13861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:45.016196 13861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:45.016284 13861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:45.016312 13861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:45.016345 13861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:45.016371 13861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:45.017948 13861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:45.018247 13861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:45.018487 13861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:45.023542 13867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:45.023535 13869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:45.023537 13866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:45.023716 13861 server_base.cc:1034] running on GCE node
I20260928 20:28:45.024015 13861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:45.024313 13861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:45.025542 13861 hybrid_clock.cc:648] HybridClock initialized: now 1790627325025525 us; error 43 us; skew 500 ppm
Sep 28 20:28:45 dist-test-slave-k477 krb5kdc[13848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:45.028275 13861 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:45.028739 13861 webserver.cc:460] Webserver started at http://127.0.0.1:44439/ using document root <none> and password file <none>
I20260928 20:28:45.028973 13861 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:45.029043 13861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:45.029206 13861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:45.030359 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "7338bdc464494394b8bc0b412fa6d7a6"
format_stamp: "Formatted at 2026-09-28 20:28:45 on dist-test-slave-k477"
server_key: "22d2f9d6583930ebee32823f9772ee7b"
server_key_iv: "0d75bc72af8839f820deb1f0a2e78daf"
server_key_version: "encryptionkey@0"
I20260928 20:28:45.030757 13861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "7338bdc464494394b8bc0b412fa6d7a6"
format_stamp: "Formatted at 2026-09-28 20:28:45 on dist-test-slave-k477"
server_key: "22d2f9d6583930ebee32823f9772ee7b"
server_key_iv: "0d75bc72af8839f820deb1f0a2e78daf"
server_key_version: "encryptionkey@0"
I20260928 20:28:45.032230 13861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:28:45.033073 13876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:45.033372 13861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:45.033483 13861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "7338bdc464494394b8bc0b412fa6d7a6"
format_stamp: "Formatted at 2026-09-28 20:28:45 on dist-test-slave-k477"
server_key: "22d2f9d6583930ebee32823f9772ee7b"
server_key_iv: "0d75bc72af8839f820deb1f0a2e78daf"
server_key_version: "encryptionkey@0"
I20260928 20:28:45.033561 13861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:45.050765 13861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:45.053069 13861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:45.053259 13861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:45.057853 13861 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33723
I20260928 20:28:45.057880 13928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33723 every 8 connection(s)
I20260928 20:28:45.058354 13861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:45.059490 13929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:45.060349 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13861
I20260928 20:28:45.060458 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:45.060693 25323 external_mini_cluster.cc:1428] Setting key 08f8d3fc72131ac1c418a815bd58c451
I20260928 20:28:45.063033 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6: Bootstrap starting.
I20260928 20:28:45.063767 13929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:45.064122 13929 log.cc:824] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:45.065043 13929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6: No bootstrap required, opened a new log
I20260928 20:28:45.067075 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:45.061301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:35043 (local address 127.0.0.1:33723)
0928 20:28:45.061523 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:45.061533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:45.061566 (+ 33us) server_negotiation.cc:407] Connection header received
0928 20:28:45.062144 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:45.062153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:45.062223 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:45.062376 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:45.062548 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:45.063895 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:45.064828 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:45.065440 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:45.065579 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:45.066327 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:45.066331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:45.066332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:45.066567 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:45.066607 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:45.066624 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:45.066713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:45.066813 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 20:28:45.066842 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260928 20:28:45.067790 13929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 20:28:45.068014 13929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:45.068081 13929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7338bdc464494394b8bc0b412fa6d7a6, State: Initialized, Role: FOLLOWER
I20260928 20:28:45.068248 13929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 20:28:45.068357 13929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:45.068394 13929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:45.068409 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:45.068446 13929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:45.069073 13929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 20:28:45.069199 13929 leader_election.cc:302] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7338bdc464494394b8bc0b412fa6d7a6; no voters:
I20260928 20:28:45.069410 13929 leader_election.cc:288] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:45.069444 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:45.069555 13934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7338bdc464494394b8bc0b412fa6d7a6, State: Running, Role: LEADER
I20260928 20:28:45.069644 13934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 20:28:45.069674 13929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:45.069973 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7338bdc464494394b8bc0b412fa6d7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260928 20:28:45.070044 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:45.070199 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7338bdc464494394b8bc0b412fa6d7a6. Latest consensus state: current_term: 1 leader_uuid: "7338bdc464494394b8bc0b412fa6d7a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7338bdc464494394b8bc0b412fa6d7a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260928 20:28:45.070333 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:45.070405 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:45.070991 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:45.072876 13944 catalog_manager.cc:1347] Generated new cluster ID: 7a0994161c1f4acd969e01e67e183dab
I20260928 20:28:45.072937 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:45.075023 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:45.071298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:38625 (local address 127.0.0.1:33723)
0928 20:28:45.071349 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:45.071351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:45.071651 (+ 300us) server_negotiation.cc:407] Connection header received
0928 20:28:45.071772 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:45.071773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:45.071784 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:45.071890 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:45.072125 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:45.072793 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:45.073684 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:45.073928 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:45.074027 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:45.074776 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:45.074777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:45.074777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:45.074783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:45.074798 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:45.074802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:45.074857 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:45.074916 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 20:28:45.074926 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:28:45.089341 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:45.089902 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:45.094013 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7338bdc464494394b8bc0b412fa6d7a6: Generated new TSK 0
I20260928 20:28:45.094233 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:45.098222 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13861
W20260928 20:28:45.102778 13846 connection.cc:535] client connection to 127.0.0.1:33723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33723): Connection reset by peer (error 104)
2026-09-28T20:28:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:45 dist-test-slave-k477 krb5kdc[13960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:45 dist-test-slave-k477 krb5kdc[13960](info): set up 2 sockets
Sep 28 20:28:45 dist-test-slave-k477 krb5kdc[13960](info): commencing operation
krb5kdc: starting...
W20260928 20:28:46.176779 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:46 dist-test-slave-k477 krb5kdc[13960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:46.303897 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:46.391059 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:46.391302 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:46.391328 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:46.393488 13973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:46.393570 13973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:46.393586 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:46.393601 13973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:46.393615 13973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:46.395542 13973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:46.395788 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:46.396083 13973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:46.401276 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:46.401384 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:46.401427 13973 server_base.cc:1034] running on GCE node
W20260928 20:28:46.401513 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:46.402098 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:46.402487 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:46.403792 13973 hybrid_clock.cc:648] HybridClock initialized: now 1790627326403794 us; error 72 us; skew 500 ppm
Sep 28 20:28:46 dist-test-slave-k477 krb5kdc[13960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:46.413318 13973 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:46.413884 13973 webserver.cc:460] Webserver started at http://127.0.0.1:46319/ using document root <none> and password file <none>
I20260928 20:28:46.414139 13973 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:46.414398 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:46.414608 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:46.415750 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c"
format_stamp: "Formatted at 2026-09-28 20:28:46 on dist-test-slave-k477"
server_key: "b6a9532122910ff7030204434a3c7bc5"
server_key_iv: "186251bb1b780831861f4f79c205e5a5"
server_key_version: "encryptionkey@0"
I20260928 20:28:46.416220 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c"
format_stamp: "Formatted at 2026-09-28 20:28:46 on dist-test-slave-k477"
server_key: "b6a9532122910ff7030204434a3c7bc5"
server_key_iv: "186251bb1b780831861f4f79c205e5a5"
server_key_version: "encryptionkey@0"
I20260928 20:28:46.417688 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:46.418627 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:46.418879 13973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:46.419000 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c"
format_stamp: "Formatted at 2026-09-28 20:28:46 on dist-test-slave-k477"
server_key: "b6a9532122910ff7030204434a3c7bc5"
server_key_iv: "186251bb1b780831861f4f79c205e5a5"
server_key_version: "encryptionkey@0"
I20260928 20:28:46.419087 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:46.422765 13973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:46.425665 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:46.425859 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:46.432502 13973 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42625
I20260928 20:28:46.432535 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42625 every 8 connection(s)
I20260928 20:28:46.432922 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:46.433632 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:46.436059 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: Bootstrap starting.
I20260928 20:28:46.436738 14041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:46.437028 14041 log.cc:824] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:46.438115 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: No bootstrap required, opened a new log
I20260928 20:28:46.439929 14041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } }
I20260928 20:28:46.440114 14041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:46.440173 14041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a4bf85efdef49ecb3a2f0eb23662b3c, State: Initialized, Role: FOLLOWER
I20260928 20:28:46.440366 14041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } }
I20260928 20:28:46.440392 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 13973
I20260928 20:28:46.440474 14041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:46.440524 14041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:46.440575 14041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:46.440596 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:46.440860 25323 external_mini_cluster.cc:1428] Setting key 9c83790b08bb25dd29282e69601651ef
I20260928 20:28:46.441339 14041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } }
I20260928 20:28:46.441473 14041 leader_election.cc:302] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a4bf85efdef49ecb3a2f0eb23662b3c; no voters:
I20260928 20:28:46.441748 14041 leader_election.cc:288] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:46.441776 14045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:46.441977 14045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [term 1 LEADER]: Becoming Leader. State: Replica: 7a4bf85efdef49ecb3a2f0eb23662b3c, State: Running, Role: LEADER
I20260928 20:28:46.442107 14045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } }
I20260928 20:28:46.442116 14041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:46.442982 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } } }
I20260928 20:28:46.443091 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:46.443786 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:46.441651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:49829 (local address 127.0.0.1:42625)
0928 20:28:46.441885 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:46.441894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:46.441956 (+ 62us) server_negotiation.cc:407] Connection header received
0928 20:28:46.442084 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:46.442094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:46.442125 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:46.442321 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:46.443095 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:46.443098 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:46.443098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:46.443103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:46.443148 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:46.443163 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:46.443221 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:46.443471 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 20:28:46.443543 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260928 20:28:46.443848 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a4bf85efdef49ecb3a2f0eb23662b3c. Latest consensus state: current_term: 1 leader_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a4bf85efdef49ecb3a2f0eb23662b3c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42625 } } }
I20260928 20:28:46.444802 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:46.444852 14062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:28:46.444952 14062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:46.445048 14051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:46.445221 14051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:46.445556 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:46.447549 14051 catalog_manager.cc:1347] Generated new cluster ID: df66493c620a4f6ba90b4a42cb4f65b9
I20260928 20:28:46.447644 14051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:46.455319 14051 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:46.455685 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:46.447166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:34601 (local address 127.0.0.1:42625)
0928 20:28:46.447210 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:46.447212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:46.447332 (+ 120us) server_negotiation.cc:407] Connection header received
0928 20:28:46.447396 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:46.447397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:46.447405 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:46.447456 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:46.455391 (+ 7935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:46.455393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:46.455393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:46.455400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:46.455421 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:46.455426 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:46.455486 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:46.455573 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 20:28:46.455591 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:28:46.455730 14051 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:46.469909 14051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a4bf85efdef49ecb3a2f0eb23662b3c: Generated new TSK 0
I20260928 20:28:46.470161 14051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:46.472981 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 13973
2026-09-28T20:28:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:46 dist-test-slave-k477 krb5kdc[14072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:46 dist-test-slave-k477 krb5kdc[14072](info): set up 2 sockets
Sep 28 20:28:46 dist-test-slave-k477 krb5kdc[14072](info): commencing operation
krb5kdc: starting...
W20260928 20:28:47.545091 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:47.687472 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:47.774796 14085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:47.775036 14085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:47.775064 14085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:47.776531 14085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:47.776602 14085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:47.776616 14085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:47.776631 14085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:47.776643 14085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:47.778299 14085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:47.778610 14085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:47.778869 14085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:47.784294 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:47.784294 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:47.784297 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:47.784734 14085 server_base.cc:1034] running on GCE node
I20260928 20:28:47.785005 14085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:47.785320 14085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:47.786517 14085 hybrid_clock.cc:648] HybridClock initialized: now 1790627327786504 us; error 31 us; skew 500 ppm
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:47.789666 14085 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:47.790218 14085 webserver.cc:460] Webserver started at http://127.0.0.1:36659/ using document root <none> and password file <none>
I20260928 20:28:47.790458 14085 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:47.790537 14085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:47.790691 14085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:47.793000 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "ff671380c6c548a1809e0a39bec89fbd"
format_stamp: "Formatted at 2026-09-28 20:28:47 on dist-test-slave-k477"
server_key: "8928059e8e11ce262a29824b8cfc8260"
server_key_iv: "0d1267f611a26dc9e7b9725319295ac1"
server_key_version: "encryptionkey@0"
I20260928 20:28:47.793545 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "ff671380c6c548a1809e0a39bec89fbd"
format_stamp: "Formatted at 2026-09-28 20:28:47 on dist-test-slave-k477"
server_key: "8928059e8e11ce262a29824b8cfc8260"
server_key_iv: "0d1267f611a26dc9e7b9725319295ac1"
server_key_version: "encryptionkey@0"
I20260928 20:28:47.795128 14085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:47.796180 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:47.796408 14085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:47.796525 14085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "ff671380c6c548a1809e0a39bec89fbd"
format_stamp: "Formatted at 2026-09-28 20:28:47 on dist-test-slave-k477"
server_key: "8928059e8e11ce262a29824b8cfc8260"
server_key_iv: "0d1267f611a26dc9e7b9725319295ac1"
server_key_version: "encryptionkey@0"
I20260928 20:28:47.796614 14085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:47.818315 14085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:47.820894 14085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:47.821090 14085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:47.825376 14085 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37149
I20260928 20:28:47.825416 14152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37149 every 8 connection(s)
I20260928 20:28:47.825799 14085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:47.827171 14153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:47.829864 14153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd: Bootstrap starting.
I20260928 20:28:47.830426 14153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:47.830679 14153 log.cc:824] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:47.831544 14153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd: No bootstrap required, opened a new log
I20260928 20:28:47.833374 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14085
I20260928 20:28:47.833505 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:47.833350 14153 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260928 20:28:47.833542 14153 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:47.833627 14153 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff671380c6c548a1809e0a39bec89fbd, State: Initialized, Role: FOLLOWER
I20260928 20:28:47.833704 25323 external_mini_cluster.cc:1428] Setting key a3022fb4a43be40c0003a861a6d6a84a
I20260928 20:28:47.833812 14153 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260928 20:28:47.833913 14153 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:47.833951 14153 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:47.834016 14153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:47.834949 14153 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260928 20:28:47.835093 14153 leader_election.cc:302] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff671380c6c548a1809e0a39bec89fbd; no voters:
I20260928 20:28:47.835294 14153 leader_election.cc:288] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:47.835353 14158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:47.835533 14158 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [term 1 LEADER]: Becoming Leader. State: Replica: ff671380c6c548a1809e0a39bec89fbd, State: Running, Role: LEADER
I20260928 20:28:47.835561 14153 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:47.835667 14158 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260928 20:28:47.836057 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff671380c6c548a1809e0a39bec89fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260928 20:28:47.836038 14158 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff671380c6c548a1809e0a39bec89fbd. Latest consensus state: current_term: 1 leader_uuid: "ff671380c6c548a1809e0a39bec89fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff671380c6c548a1809e0a39bec89fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260928 20:28:47.836232 14158 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:47.836222 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:47.836438 14167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:47.836967 14167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:47.838443 14167 catalog_manager.cc:1347] Generated new cluster ID: 3dc756aac9b448e6af3be75f9559c79e
I20260928 20:28:47.838496 14167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 20:28:47.847244 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:47.834564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:46741 (local address 127.0.0.1:37149)
0928 20:28:47.835223 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:47.835235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:47.835273 (+ 38us) server_negotiation.cc:407] Connection header received
0928 20:28:47.835407 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:47.835415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:47.835478 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:47.836583 (+ 1105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:47.836723 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:47.838365 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:47.839477 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:47.840136 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:47.840252 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:47.843212 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:47.843217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:47.843217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:47.843231 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:47.845297 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:47.845847 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:47.845851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:47.845852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:47.845984 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:47.846259 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:47.846260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:47.846260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:47.846629 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:47.846813 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:47.846896 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 20:28:47.847006 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":552,"threads_started":1}
I20260928 20:28:47.848336 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:47.857800 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:47.850083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:37559 (local address 127.0.0.1:37149)
0928 20:28:47.850140 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:47.850142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:47.850488 (+ 346us) server_negotiation.cc:407] Connection header received
0928 20:28:47.850522 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:47.850523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:47.850534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:47.850626 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:47.851241 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:47.852073 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:47.853076 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:47.853429 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:47.853512 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:47.854907 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:47.854909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:47.854909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:47.854918 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:47.856568 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:47.856978 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:47.856979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:47.856979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:47.857042 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:47.857277 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:47.857278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:47.857278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:47.857418 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:47.857534 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:47.857637 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 20:28:47.857701 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:28:47.866564 14167 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:47.867182 14167 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:47.874239 14167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff671380c6c548a1809e0a39bec89fbd: Generated new TSK 0
I20260928 20:28:47.874457 14167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:47.878795 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14085
W20260928 20:28:47.883313 14070 connection.cc:535] client connection to 127.0.0.1:37149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37149): Connection reset by peer (error 104)
2026-09-28T20:28:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14184](info): set up 2 sockets
Sep 28 20:28:47 dist-test-slave-k477 krb5kdc[14184](info): commencing operation
krb5kdc: starting...
W20260928 20:28:48.932653 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:49 dist-test-slave-k477 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:49.063737 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:49.153963 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:49.154199 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:49.154223 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:49.155342 14197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:49.155385 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:49.155400 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:49.155412 14197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:49.155424 14197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:49.158145 14197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:49.158483 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:49.158773 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:49.164088 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:49.164413 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:49.164433 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:49.164839 14197 server_base.cc:1034] running on GCE node
I20260928 20:28:49.165083 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:49.165539 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:49.166739 14197 hybrid_clock.cc:648] HybridClock initialized: now 1790627329166739 us; error 43 us; skew 500 ppm
Sep 28 20:28:49 dist-test-slave-k477 krb5kdc[14184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:49.169708 14197 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:49.170483 14197 webserver.cc:460] Webserver started at http://127.0.0.1:36245/ using document root <none> and password file <none>
I20260928 20:28:49.170737 14197 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:49.170809 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:49.170964 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:49.172188 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "be7a9c72144345d488f544534428b8e1"
format_stamp: "Formatted at 2026-09-28 20:28:49 on dist-test-slave-k477"
server_key: "4e000f040c669f418bd848607253282d"
server_key_iv: "04758d21dd4faf65e0163f040b9126e6"
server_key_version: "encryptionkey@0"
I20260928 20:28:49.172550 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "be7a9c72144345d488f544534428b8e1"
format_stamp: "Formatted at 2026-09-28 20:28:49 on dist-test-slave-k477"
server_key: "4e000f040c669f418bd848607253282d"
server_key_iv: "04758d21dd4faf65e0163f040b9126e6"
server_key_version: "encryptionkey@0"
I20260928 20:28:49.173959 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:49.174990 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:49.175249 14197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:49.175357 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "be7a9c72144345d488f544534428b8e1"
format_stamp: "Formatted at 2026-09-28 20:28:49 on dist-test-slave-k477"
server_key: "4e000f040c669f418bd848607253282d"
server_key_iv: "04758d21dd4faf65e0163f040b9126e6"
server_key_version: "encryptionkey@0"
I20260928 20:28:49.175449 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:49.196938 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:49.199455 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:49.199702 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:49.203652 14197 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33369
I20260928 20:28:49.203686 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33369 every 8 connection(s)
I20260928 20:28:49.204099 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:49.204999 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:49.207273 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: Bootstrap starting.
I20260928 20:28:49.207926 14265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:49.208218 14265 log.cc:824] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:49.208999 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: No bootstrap required, opened a new log
I20260928 20:28:49.210347 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14197
I20260928 20:28:49.210465 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:49.210721 25323 external_mini_cluster.cc:1428] Setting key 642a252e264cb56ba1f2624a58790207
I20260928 20:28:49.210844 14265 raft_consensus.cc:348] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260928 20:28:49.211071 14265 raft_consensus.cc:374] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:49.211119 14265 raft_consensus.cc:729] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7a9c72144345d488f544534428b8e1, State: Initialized, Role: FOLLOWER
I20260928 20:28:49.211270 14265 consensus_queue.cc:260] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260928 20:28:49.211351 14265 raft_consensus.cc:388] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:49.211386 14265 raft_consensus.cc:482] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:49.211434 14265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:49.212554 14265 raft_consensus.cc:504] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260928 20:28:49.212661 14265 leader_election.cc:302] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be7a9c72144345d488f544534428b8e1; no voters:
I20260928 20:28:49.212845 14265 leader_election.cc:288] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:49.212888 14270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:49.213006 14270 raft_consensus.cc:686] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [term 1 LEADER]: Becoming Leader. State: Replica: be7a9c72144345d488f544534428b8e1, State: Running, Role: LEADER
I20260928 20:28:49.213122 14265 sys_catalog.cc:564] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:49.213097 14270 consensus_queue.cc:237] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } }
I20260928 20:28:49.213460 14270 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be7a9c72144345d488f544534428b8e1. Latest consensus state: current_term: 1 leader_uuid: "be7a9c72144345d488f544534428b8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260928 20:28:49.213562 14270 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:49.213465 14272 sys_catalog.cc:455] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be7a9c72144345d488f544534428b8e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be7a9c72144345d488f544534428b8e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33369 } } }
I20260928 20:28:49.213689 14272 sys_catalog.cc:458] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1 [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:49.215083 14286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:28:49.215138 14286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:49.215181 14276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:49.215309 14276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:49.216920 14276 catalog_manager.cc:1347] Generated new cluster ID: 9a6b20f6e9a84066b8dc85c38a9df1ea
I20260928 20:28:49.216979 14276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:49.217316 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:49.211591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:48835 (local address 127.0.0.1:33369)
0928 20:28:49.212002 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:49.212010 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:49.212044 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:28:49.212149 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:49.212155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:49.212204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:49.212323 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:49.212567 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:49.213923 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:49.215004 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:49.215573 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:49.215690 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:49.216604 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:49.216607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:49.216607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:49.216769 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:49.216800 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:49.216833 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:49.216917 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:49.217120 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 20:28:49.217154 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":296,"threads_started":1}
I20260928 20:28:49.220958 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:49.225564 14269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:49.222433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:37243 (local address 127.0.0.1:33369)
0928 20:28:49.222476 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:49.222479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:49.222605 (+ 126us) server_negotiation.cc:407] Connection header received
0928 20:28:49.222685 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:49.222686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:49.222706 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:49.222757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:49.222912 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:49.223437 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:49.224246 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:49.224486 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:49.224566 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:49.225276 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:49.225277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:49.225277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:49.225282 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:49.225299 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:49.225304 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:49.225362 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:49.225438 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 20:28:49.225452 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:28:49.240718 14276 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:49.241256 14276 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:49.246533 14276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be7a9c72144345d488f544534428b8e1: Generated new TSK 0
I20260928 20:28:49.246758 14276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:49.253502 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14197
W20260928 20:28:49.258152 14182 connection.cc:535] client connection to 127.0.0.1:33369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33369): Connection reset by peer (error 104)
2026-09-28T20:28:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:49 dist-test-slave-k477 krb5kdc[14296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:49 dist-test-slave-k477 krb5kdc[14296](info): set up 2 sockets
Sep 28 20:28:49 dist-test-slave-k477 krb5kdc[14296](info): commencing operation
krb5kdc: starting...
W20260928 20:28:50.324666 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:50 dist-test-slave-k477 krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 20:28:50.457072 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:50.544029 14309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:50.544306 14309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:50.544349 14309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:50.545598 14309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:50.545683 14309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:50.545709 14309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:50.545737 14309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:50.545769 14309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:50.547670 14309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39623
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:50.548025 14309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:50.548288 14309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:50.553753 14317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:50.553843 14309 server_base.cc:1034] running on GCE node
W20260928 20:28:50.553727 14315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:50.553727 14314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:50.554282 14309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:50.554817 14309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:50.556020 14309 hybrid_clock.cc:648] HybridClock initialized: now 1790627330556004 us; error 41 us; skew 500 ppm
Sep 28 20:28:50 dist-test-slave-k477 krb5kdc[14296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:50.558772 14309 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 20:28:50.559293 14309 webserver.cc:460] Webserver started at http://127.0.0.1:35993/ using document root <none> and password file <none>
I20260928 20:28:50.559525 14309 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:50.559600 14309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:50.559754 14309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:50.560868 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "57dac6d2ad1f42419862f0ffc877e24d"
format_stamp: "Formatted at 2026-09-28 20:28:50 on dist-test-slave-k477"
server_key: "14b12254d1cab6f0f1ea92e50c28ed4c"
server_key_iv: "8ca1c718e410ab051be403f29f962c3a"
server_key_version: "encryptionkey@0"
I20260928 20:28:50.561244 14309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "57dac6d2ad1f42419862f0ffc877e24d"
format_stamp: "Formatted at 2026-09-28 20:28:50 on dist-test-slave-k477"
server_key: "14b12254d1cab6f0f1ea92e50c28ed4c"
server_key_iv: "8ca1c718e410ab051be403f29f962c3a"
server_key_version: "encryptionkey@0"
I20260928 20:28:50.562644 14309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:50.563733 14324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:50.564103 14309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:50.564256 14309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "57dac6d2ad1f42419862f0ffc877e24d"
format_stamp: "Formatted at 2026-09-28 20:28:50 on dist-test-slave-k477"
server_key: "14b12254d1cab6f0f1ea92e50c28ed4c"
server_key_iv: "8ca1c718e410ab051be403f29f962c3a"
server_key_version: "encryptionkey@0"
I20260928 20:28:50.564380 14309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:50.568145 14309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:50.570344 14309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:50.570547 14309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:50.575093 14309 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39623
I20260928 20:28:50.575203 14376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39623 every 8 connection(s)
I20260928 20:28:50.575568 14309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:50.576994 14377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:50.579310 14377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d: Bootstrap starting.
I20260928 20:28:50.579962 14377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:50.580201 14377 log.cc:824] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:50.580901 14377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d: No bootstrap required, opened a new log
I20260928 20:28:50.582417 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14309
I20260928 20:28:50.582568 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:50.582477 14377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260928 20:28:50.582799 25323 external_mini_cluster.cc:1428] Setting key 3e9b087efbe09cdadbc0b8cf2602c766
I20260928 20:28:50.582803 14377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:50.582880 14377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dac6d2ad1f42419862f0ffc877e24d, State: Initialized, Role: FOLLOWER
I20260928 20:28:50.583068 14377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260928 20:28:50.583134 14377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:50.583156 14377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:50.583184 14377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:50.584462 14377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260928 20:28:50.584595 14377 leader_election.cc:302] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57dac6d2ad1f42419862f0ffc877e24d; no voters:
I20260928 20:28:50.584776 14377 leader_election.cc:288] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:50.584847 14382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:50.585052 14382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [term 1 LEADER]: Becoming Leader. State: Replica: 57dac6d2ad1f42419862f0ffc877e24d, State: Running, Role: LEADER
I20260928 20:28:50.585093 14377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:50.585258 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:50.583463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:47969 (local address 127.0.0.1:39623)
0928 20:28:50.583678 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:50.583686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:50.583712 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:28:50.583877 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:50.583884 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:50.583909 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:50.584023 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:50.584695 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:50.584699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:50.584699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:50.584705 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:50.584752 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:50.584768 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:50.584837 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:50.584936 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 20:28:50.584983 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260928 20:28:50.585222 14382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } }
I20260928 20:28:50.585601 14382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57dac6d2ad1f42419862f0ffc877e24d. Latest consensus state: current_term: 1 leader_uuid: "57dac6d2ad1f42419862f0ffc877e24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } } }
I20260928 20:28:50.585745 14382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:50.585603 14383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57dac6d2ad1f42419862f0ffc877e24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57dac6d2ad1f42419862f0ffc877e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39623 } } }
I20260928 20:28:50.585991 14391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:50.586058 14383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:50.586242 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:50.586632 14391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:50.588892 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:50.587792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:51395 (local address 127.0.0.1:39623)
0928 20:28:50.587848 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:50.587850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:50.587996 (+ 146us) server_negotiation.cc:407] Connection header received
0928 20:28:50.588040 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:50.588040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:50.588046 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:50.588102 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:50.588697 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:50.588698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:50.588698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:50.588703 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:50.588717 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:50.588722 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:50.588787 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:50.588821 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 20:28:50.588833 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:28:50.593585 14391 catalog_manager.cc:1347] Generated new cluster ID: c95e73a1f5ce4c7b8f230520aaca3509
I20260928 20:28:50.593662 14391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:50.605994 14391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:50.606204 14391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:50.609954 14391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57dac6d2ad1f42419862f0ffc877e24d: Generated new TSK 0
I20260928 20:28:50.610172 14391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:50.616380 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14309
2026-09-28T20:28:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:50 dist-test-slave-k477 krb5kdc[14408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:50 dist-test-slave-k477 krb5kdc[14408](info): set up 2 sockets
Sep 28 20:28:50 dist-test-slave-k477 krb5kdc[14408](info): commencing operation
krb5kdc: starting...
W20260928 20:28:51.696610 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:51 dist-test-slave-k477 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:51.826493 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38879
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:32805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:51.912410 14421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:51.912622 14421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:51.912647 14421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:51.913885 14421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:51.913944 14421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:51.913959 14421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:51.913970 14421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:51.913982 14421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:51.915582 14421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:32805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:51.915915 14421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:51.916177 14421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:51.921350 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:51.921415 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:51.921350 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:51.921613 14421 server_base.cc:1034] running on GCE node
I20260928 20:28:51.921860 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:51.922176 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:51.925450 14421 hybrid_clock.cc:648] HybridClock initialized: now 1790627331924802 us; error 659 us; skew 500 ppm
Sep 28 20:28:51 dist-test-slave-k477 krb5kdc[14408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627331, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:51.934965 14421 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:51.935658 14421 webserver.cc:460] Webserver started at http://127.24.186.254:44489/ using document root <none> and password file <none>
I20260928 20:28:51.935984 14421 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:51.936039 14421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:51.936242 14421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:51.937460 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "d33dcc41a0a5479285e453cf29e78b6a"
format_stamp: "Formatted at 2026-09-28 20:28:51 on dist-test-slave-k477"
server_key: "4c4cb76d171098df19d6a219919ff408"
server_key_iv: "aba94507387b9eedf900697e1e34219d"
server_key_version: "encryptionkey@0"
I20260928 20:28:51.937840 14421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "d33dcc41a0a5479285e453cf29e78b6a"
format_stamp: "Formatted at 2026-09-28 20:28:51 on dist-test-slave-k477"
server_key: "4c4cb76d171098df19d6a219919ff408"
server_key_iv: "aba94507387b9eedf900697e1e34219d"
server_key_version: "encryptionkey@0"
I20260928 20:28:51.939258 14421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 20:28:51.940147 14436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:51.940399 14421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:51.940492 14421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "d33dcc41a0a5479285e453cf29e78b6a"
format_stamp: "Formatted at 2026-09-28 20:28:51 on dist-test-slave-k477"
server_key: "4c4cb76d171098df19d6a219919ff408"
server_key_iv: "aba94507387b9eedf900697e1e34219d"
server_key_version: "encryptionkey@0"
I20260928 20:28:51.940590 14421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:51.953788 14421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:51.955950 14421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:51.956123 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:51.959913 14421 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38879
I20260928 20:28:51.959991 14488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38879 every 8 connection(s)
I20260928 20:28:51.960902 14421 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:51.962109 14489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:51.962422 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14421
I20260928 20:28:51.962572 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:51.962788 25323 external_mini_cluster.cc:1428] Setting key 66669d473d3ab2f533fc8833bbb5de22
I20260928 20:28:51.965458 14489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: Bootstrap starting.
I20260928 20:28:51.966094 14489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:51.966388 14489 log.cc:824] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:51.967096 14489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: No bootstrap required, opened a new log
I20260928 20:28:51.969311 14489 raft_consensus.cc:348] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } }
I20260928 20:28:51.969521 14489 raft_consensus.cc:374] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:51.969569 14489 raft_consensus.cc:729] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33dcc41a0a5479285e453cf29e78b6a, State: Initialized, Role: FOLLOWER
Sep 28 20:28:51 dist-test-slave-k477 krb5kdc[14408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:51.969704 14489 consensus_queue.cc:260] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } }
I20260928 20:28:51.969784 14489 raft_consensus.cc:388] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:51.969820 14489 raft_consensus.cc:482] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:51.969866 14489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:51.970525 14489 raft_consensus.cc:504] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } }
I20260928 20:28:51.970659 14489 leader_election.cc:302] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d33dcc41a0a5479285e453cf29e78b6a; no voters:
I20260928 20:28:51.970893 14489 leader_election.cc:288] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:51.970942 14494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:51.971195 14489 sys_catalog.cc:564] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:51.971391 14494 raft_consensus.cc:686] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [term 1 LEADER]: Becoming Leader. State: Replica: d33dcc41a0a5479285e453cf29e78b6a, State: Running, Role: LEADER
I20260928 20:28:51.971626 14494 consensus_queue.cc:237] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } }
I20260928 20:28:51.972939 14496 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d33dcc41a0a5479285e453cf29e78b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } } }
I20260928 20:28:51.973363 14496 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:51.973907 14508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 20:28:51.973085 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d33dcc41a0a5479285e453cf29e78b6a. Latest consensus state: current_term: 1 leader_uuid: "d33dcc41a0a5479285e453cf29e78b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d33dcc41a0a5479285e453cf29e78b6a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38879 } } }
I20260928 20:28:51.974056 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:51.973999 14508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:51.974238 14509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:51.974525 14509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:51.975449 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:51.963366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34837 (local address 127.24.186.254:38879)
0928 20:28:51.963563 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:51.963571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:51.963598 (+ 27us) server_negotiation.cc:407] Connection header received
0928 20:28:51.964102 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:51.964109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:51.964159 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:51.964282 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:51.965121 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:51.966094 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:51.966940 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:51.967401 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:51.967550 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:51.970619 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:51.970626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:51.970626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:51.970666 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:51.973030 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:51.973919 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:51.973923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:51.973923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:51.974056 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:51.974421 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:51.974422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:51.974422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:51.974782 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:51.974966 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:51.975128 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 20:28:51.975243 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260928 20:28:51.976502 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:51.977747 14509 catalog_manager.cc:1347] Generated new cluster ID: 2972e0a51d3c43b8bc01b3b9860189d7
I20260928 20:28:51.977833 14509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:51.987016 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:51.977949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36103 (local address 127.24.186.254:38879)
0928 20:28:51.977987 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:51.977989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:51.978021 (+ 32us) server_negotiation.cc:407] Connection header received
0928 20:28:51.978098 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:51.978098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:51.978109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:51.978180 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:51.978794 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:51.979948 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:51.981579 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:51.981940 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:51.982050 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:51.984001 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:51.984003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:51.984003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:51.984011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:51.985728 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:51.986171 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:51.986172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:51.986173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:51.986236 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:51.986464 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:51.986465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:51.986466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:51.986608 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:51.986728 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:51.986841 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 20:28:51.986912 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:28:51.998086 14509 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:51.998677 14509 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:52.004513 14509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d33dcc41a0a5479285e453cf29e78b6a: Generated new TSK 0
I20260928 20:28:52.004757 14509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:52.015745 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14421
W20260928 20:28:52.021443 14406 connection.cc:535] client connection to 127.24.186.254:38879 recv error: Network error: failed to read from TLS socket (remote: 127.24.186.254:38879): Connection reset by peer (error 104)
2026-09-28T20:28:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:52 dist-test-slave-k477 krb5kdc[14520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:52 dist-test-slave-k477 krb5kdc[14520](info): set up 2 sockets
Sep 28 20:28:52 dist-test-slave-k477 krb5kdc[14520](info): commencing operation
krb5kdc: starting...
W20260928 20:28:53.080935 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:53 dist-test-slave-k477 krb5kdc[14520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:53.208495 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38133
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:53.291476 14533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:53.291689 14533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:53.291713 14533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:53.293128 14533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:53.293195 14533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:53.293211 14533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:53.293224 14533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:53.293236 14533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:53.295095 14533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:53.295329 14533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:53.295518 14533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:53.300467 14538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:53.300509 14541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:53.300514 14539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:53.301327 14533 server_base.cc:1034] running on GCE node
I20260928 20:28:53.301776 14533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:53.302096 14533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:53.303316 14533 hybrid_clock.cc:648] HybridClock initialized: now 1790627333303291 us; error 46 us; skew 500 ppm
Sep 28 20:28:53 dist-test-slave-k477 krb5kdc[14520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627333, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:53.305926 14533 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:53.306388 14533 webserver.cc:460] Webserver started at http://127.24.186.254:35207/ using document root <none> and password file <none>
I20260928 20:28:53.306644 14533 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:53.306715 14533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:53.306864 14533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:53.307997 14533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "a6ea22f92c3f403a8a6254d86f67bc5f"
format_stamp: "Formatted at 2026-09-28 20:28:53 on dist-test-slave-k477"
server_key: "4f36893c11ec02b60f1e5a94eb4bcb88"
server_key_iv: "bffc5f0c82dbb12f426679344bca4e03"
server_key_version: "encryptionkey@0"
I20260928 20:28:53.308363 14533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "a6ea22f92c3f403a8a6254d86f67bc5f"
format_stamp: "Formatted at 2026-09-28 20:28:53 on dist-test-slave-k477"
server_key: "4f36893c11ec02b60f1e5a94eb4bcb88"
server_key_iv: "bffc5f0c82dbb12f426679344bca4e03"
server_key_version: "encryptionkey@0"
I20260928 20:28:53.309710 14533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:28:53.310463 14548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:53.310657 14533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:53.310765 14533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "a6ea22f92c3f403a8a6254d86f67bc5f"
format_stamp: "Formatted at 2026-09-28 20:28:53 on dist-test-slave-k477"
server_key: "4f36893c11ec02b60f1e5a94eb4bcb88"
server_key_iv: "bffc5f0c82dbb12f426679344bca4e03"
server_key_version: "encryptionkey@0"
I20260928 20:28:53.310842 14533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:53.320876 14533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:53.323268 14533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:53.323478 14533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:53.327551 14533 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38133
I20260928 20:28:53.327605 14600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38133 every 8 connection(s)
I20260928 20:28:53.327999 14533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:53.328862 14601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:53.332005 14601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f: Bootstrap starting.
I20260928 20:28:53.332549 14601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:53.332774 14601 log.cc:824] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:53.333425 14601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f: No bootstrap required, opened a new log
I20260928 20:28:53.334205 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14533
I20260928 20:28:53.334318 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:53.334497 25323 external_mini_cluster.cc:1428] Setting key 651ca3163bc6289c253470bec161e1a2
I20260928 20:28:53.335564 14601 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } }
I20260928 20:28:53.335734 14601 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:53.335773 14601 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6ea22f92c3f403a8a6254d86f67bc5f, State: Initialized, Role: FOLLOWER
I20260928 20:28:53.335934 14601 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } }
I20260928 20:28:53.336027 14601 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:53.336066 14601 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:53.336159 14601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:53.336922 14601 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } }
I20260928 20:28:53.337030 14601 leader_election.cc:302] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6ea22f92c3f403a8a6254d86f67bc5f; no voters:
I20260928 20:28:53.337181 14601 leader_election.cc:288] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:53.337416 14601 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:53.337740 14606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:53.338001 14606 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [term 1 LEADER]: Becoming Leader. State: Replica: a6ea22f92c3f403a8a6254d86f67bc5f, State: Running, Role: LEADER
I20260928 20:28:53.338132 14606 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } }
I20260928 20:28:53.338891 14606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6ea22f92c3f403a8a6254d86f67bc5f. Latest consensus state: current_term: 1 leader_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } } }
I20260928 20:28:53.338984 14608 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6ea22f92c3f403a8a6254d86f67bc5f" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38133 } } }
I20260928 20:28:53.339043 14608 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:53.339028 14606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:53.339347 14619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:53.339939 14619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:53.341634 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:53.335032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56901 (local address 127.24.186.254:38133)
0928 20:28:53.335224 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:53.335240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:53.335266 (+ 26us) server_negotiation.cc:407] Connection header received
0928 20:28:53.335540 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:53.335546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:53.335591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:53.335706 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:53.336013 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:53.337859 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:53.339120 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:53.339767 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:53.339906 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:53.340862 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:53.340867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:53.340867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:53.341073 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:53.341112 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:53.341127 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:53.341214 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:53.341365 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 20:28:53.341398 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260928 20:28:53.341641 14619 catalog_manager.cc:1347] Generated new cluster ID: 9751f8c57f6a4330b344194d6697c2fd
I20260928 20:28:53.341737 14619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:53.342626 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:53.347055 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:53.343940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34035 (local address 127.24.186.254:38133)
0928 20:28:53.343976 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:53.343978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:53.344177 (+ 199us) server_negotiation.cc:407] Connection header received
0928 20:28:53.344248 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:53.344249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:53.344274 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:53.344323 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:53.344447 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:53.344914 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:53.345627 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:53.345863 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:53.345940 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:53.346749 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:53.346750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:53.346751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:53.346756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:53.346775 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:53.346780 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:53.346838 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:53.346935 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 20:28:53.346950 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 20:28:53.350159 14619 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:53.350684 14619 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:53.355572 14619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6ea22f92c3f403a8a6254d86f67bc5f: Generated new TSK 0
I20260928 20:28:53.355772 14619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:53.359292 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14533
W20260928 20:28:53.363335 14518 connection.cc:535] client connection to 127.24.186.254:38133 recv error: Network error: failed to read from TLS socket (remote: 127.24.186.254:38133): Connection reset by peer (error 104)
2026-09-28T20:28:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:53 dist-test-slave-k477 krb5kdc[14632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:53 dist-test-slave-k477 krb5kdc[14632](info): set up 2 sockets
Sep 28 20:28:53 dist-test-slave-k477 krb5kdc[14632](info): commencing operation
krb5kdc: starting...
W20260928 20:28:54.412914 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:54 dist-test-slave-k477 krb5kdc[14632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:54.547379 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46217
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:54.635156 14645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:54.635429 14645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:54.635473 14645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:54.636708 14645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:54.636782 14645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:54.636808 14645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:54.636862 14645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:54.636890 14645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:54.638407 14645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:46217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:54.638712 14645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:54.638950 14645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:54.644730 14653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:54.644771 14650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:54.644730 14651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:54.645013 14645 server_base.cc:1034] running on GCE node
I20260928 20:28:54.645258 14645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:54.645511 14645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:54.646665 14645 hybrid_clock.cc:648] HybridClock initialized: now 1790627334646630 us; error 49 us; skew 500 ppm
Sep 28 20:28:54 dist-test-slave-k477 krb5kdc[14632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627334, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:54.649435 14645 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:54.649969 14645 webserver.cc:460] Webserver started at http://127.24.186.254:36189/ using document root <none> and password file <none>
I20260928 20:28:54.650236 14645 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:54.650306 14645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:54.650457 14645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:54.651543 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d"
format_stamp: "Formatted at 2026-09-28 20:28:54 on dist-test-slave-k477"
server_key: "f1dcef42748c6827bc9b4b525c3fc59a"
server_key_iv: "8e83fd8be23c490a321537d2b510c9fd"
server_key_version: "encryptionkey@0"
I20260928 20:28:54.651965 14645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d"
format_stamp: "Formatted at 2026-09-28 20:28:54 on dist-test-slave-k477"
server_key: "f1dcef42748c6827bc9b4b525c3fc59a"
server_key_iv: "8e83fd8be23c490a321537d2b510c9fd"
server_key_version: "encryptionkey@0"
I20260928 20:28:54.653375 14645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:54.654209 14660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:54.654428 14645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:54.654542 14645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d"
format_stamp: "Formatted at 2026-09-28 20:28:54 on dist-test-slave-k477"
server_key: "f1dcef42748c6827bc9b4b525c3fc59a"
server_key_iv: "8e83fd8be23c490a321537d2b510c9fd"
server_key_version: "encryptionkey@0"
I20260928 20:28:54.654630 14645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:54.658053 14645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:54.660373 14645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:54.660547 14645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:54.664875 14645 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:46217
I20260928 20:28:54.664904 14712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:46217 every 8 connection(s)
I20260928 20:28:54.665251 14645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:54.666237 14713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:54.668639 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: Bootstrap starting.
I20260928 20:28:54.669234 14713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:54.669526 14713 log.cc:824] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:54.670264 14713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: No bootstrap required, opened a new log
I20260928 20:28:54.671968 14713 raft_consensus.cc:348] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } }
I20260928 20:28:54.672127 14713 raft_consensus.cc:374] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:54.672151 14713 raft_consensus.cc:729] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea0f7e0a1df4a73ab05bc541efb0c9d, State: Initialized, Role: FOLLOWER
I20260928 20:28:54.672262 14713 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } }
I20260928 20:28:54.672307 14713 raft_consensus.cc:388] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:54.672328 14713 raft_consensus.cc:482] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:54.672355 14713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:54.672991 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14645
I20260928 20:28:54.673156 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:54.673390 25323 external_mini_cluster.cc:1428] Setting key dbf6c5685ea6420d96b161787615efb0
I20260928 20:28:54.673645 14713 raft_consensus.cc:504] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } }
I20260928 20:28:54.673779 14713 leader_election.cc:302] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea0f7e0a1df4a73ab05bc541efb0c9d; no voters:
I20260928 20:28:54.673955 14713 leader_election.cc:288] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:54.674274 14713 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:54.673978 14717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:54.674908 14717 raft_consensus.cc:686] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [term 1 LEADER]: Becoming Leader. State: Replica: cea0f7e0a1df4a73ab05bc541efb0c9d, State: Running, Role: LEADER
I20260928 20:28:54.675103 14717 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } }
I20260928 20:28:54.675796 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:54.673878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37751 (local address 127.24.186.254:46217)
0928 20:28:54.674130 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:54.674141 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:54.674180 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:28:54.674303 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:54.674311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:54.674344 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:54.674503 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:54.675216 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:54.675219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:54.675220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:54.675225 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:54.675266 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:54.675281 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:54.675459 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:54.675534 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 20:28:54.675578 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":114,"threads_started":1}
I20260928 20:28:54.675830 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } } }
I20260928 20:28:54.675850 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea0f7e0a1df4a73ab05bc541efb0c9d. Latest consensus state: current_term: 1 leader_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea0f7e0a1df4a73ab05bc541efb0c9d" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 46217 } } }
I20260928 20:28:54.676168 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:54.676191 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:54.676983 14732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 20:28:54.677013 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 20:28:54.677062 14732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:54.677134 14733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:54.677287 14733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:54.678874 14733 catalog_manager.cc:1347] Generated new cluster ID: 8e47dd693c3e48dfb98889d5c66ffbe5
I20260928 20:28:54.678930 14733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:54.679926 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:54.678530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35751 (local address 127.24.186.254:46217)
0928 20:28:54.678569 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:54.678571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:54.678698 (+ 127us) server_negotiation.cc:407] Connection header received
0928 20:28:54.678777 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:54.678777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:54.678784 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:54.678841 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:54.679577 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:54.679578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:54.679578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:54.679583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:54.679600 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:54.679605 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:54.679657 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:54.679754 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 20:28:54.679768 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 20:28:54.713802 14733 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:54.714015 14733 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:54.729260 14733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cea0f7e0a1df4a73ab05bc541efb0c9d: Generated new TSK 0
I20260928 20:28:54.729594 14733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:54.738396 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14645
2026-09-28T20:28:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:54 dist-test-slave-k477 krb5kdc[14744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:54 dist-test-slave-k477 krb5kdc[14744](info): set up 2 sockets
Sep 28 20:28:54 dist-test-slave-k477 krb5kdc[14744](info): commencing operation
krb5kdc: starting...
W20260928 20:28:55.800709 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:55 dist-test-slave-k477 krb5kdc[14744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:55.933115 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:37583
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:37583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:56.014210 14757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:56.014479 14757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:56.014526 14757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:56.015715 14757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:56.015802 14757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:56.015851 14757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:56.015946 14757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:56.015978 14757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:56.017544 14757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:37583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:56.017840 14757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:56.018080 14757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:56.022913 14763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:56.022964 14765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:56.022980 14762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:56.023298 14757 server_base.cc:1034] running on GCE node
I20260928 20:28:56.023587 14757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:56.023942 14757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:56.025100 14757 hybrid_clock.cc:648] HybridClock initialized: now 1790627336025063 us; error 51 us; skew 500 ppm
Sep 28 20:28:56 dist-test-slave-k477 krb5kdc[14744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627336, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:56.027788 14757 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:56.028373 14757 webserver.cc:460] Webserver started at http://127.24.186.254:33569/ using document root <none> and password file <none>
I20260928 20:28:56.028615 14757 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:56.028695 14757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:56.028851 14757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:56.029961 14757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "9f2dbbb085db45659cd1438b4d60d82a"
format_stamp: "Formatted at 2026-09-28 20:28:56 on dist-test-slave-k477"
server_key: "7cca0c90e90f7cacc30813681c956094"
server_key_iv: "ac60cedb88e21ab7bedfbd4cf6f82c8e"
server_key_version: "encryptionkey@0"
I20260928 20:28:56.030337 14757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "9f2dbbb085db45659cd1438b4d60d82a"
format_stamp: "Formatted at 2026-09-28 20:28:56 on dist-test-slave-k477"
server_key: "7cca0c90e90f7cacc30813681c956094"
server_key_iv: "ac60cedb88e21ab7bedfbd4cf6f82c8e"
server_key_version: "encryptionkey@0"
I20260928 20:28:56.031873 14757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 20:28:56.032745 14772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:56.032914 14757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 20:28:56.033017 14757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9f2dbbb085db45659cd1438b4d60d82a"
format_stamp: "Formatted at 2026-09-28 20:28:56 on dist-test-slave-k477"
server_key: "7cca0c90e90f7cacc30813681c956094"
server_key_iv: "ac60cedb88e21ab7bedfbd4cf6f82c8e"
server_key_version: "encryptionkey@0"
I20260928 20:28:56.033111 14757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:56.044699 14757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:56.047111 14757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:56.047289 14757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:56.051519 14757 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:37583
I20260928 20:28:56.051574 14824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:37583 every 8 connection(s)
I20260928 20:28:56.052014 14757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:56.052858 14825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:56.055042 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a: Bootstrap starting.
I20260928 20:28:56.055572 14825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:56.055884 14825 log.cc:824] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:56.056661 14825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a: No bootstrap required, opened a new log
I20260928 20:28:56.058408 14825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } }
I20260928 20:28:56.058578 14825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:56.058604 14825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2dbbb085db45659cd1438b4d60d82a, State: Initialized, Role: FOLLOWER
I20260928 20:28:56.058770 14825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } }
I20260928 20:28:56.058866 14825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:56.058893 14825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:56.058925 14825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:56.059250 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14757
I20260928 20:28:56.059381 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:56.059569 25323 external_mini_cluster.cc:1428] Setting key 56e026bac3255686e922394236bf4abe
I20260928 20:28:56.060194 14825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } }
I20260928 20:28:56.060410 14825 leader_election.cc:302] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2dbbb085db45659cd1438b4d60d82a; no voters:
I20260928 20:28:56.060617 14825 leader_election.cc:288] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:56.060704 14830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:56.060871 14830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [term 1 LEADER]: Becoming Leader. State: Replica: 9f2dbbb085db45659cd1438b4d60d82a, State: Running, Role: LEADER
I20260928 20:28:56.060920 14825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:56.060963 14830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } }
I20260928 20:28:56.061335 14832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2dbbb085db45659cd1438b4d60d82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } } }
I20260928 20:28:56.061450 14832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:56.061335 14831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2dbbb085db45659cd1438b4d60d82a. Latest consensus state: current_term: 1 leader_uuid: "9f2dbbb085db45659cd1438b4d60d82a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2dbbb085db45659cd1438b4d60d82a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 37583 } } }
I20260928 20:28:56.061599 14831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:56.062292 14835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:56.062904 14835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:56.064798 14835 catalog_manager.cc:1347] Generated new cluster ID: 5c2fa92d5a044952a03c0fa019ffa555
I20260928 20:28:56.064862 14835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 20:28:56 dist-test-slave-k477 krb5kdc[14744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:28:56.073719 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:56.060128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44541 (local address 127.24.186.254:37583)
0928 20:28:56.061383 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:56.061393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:56.061428 (+ 35us) server_negotiation.cc:407] Connection header received
0928 20:28:56.061597 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:56.061607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:56.061682 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:56.061911 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:56.062746 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:56.064332 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:56.065456 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:56.066244 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:56.066374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:56.069709 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:56.069715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:56.069716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:56.069730 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:56.071990 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:56.072418 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:56.072421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:56.072421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:56.072546 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:56.072767 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:56.072768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:56.072768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:56.073086 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:56.073241 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:56.073357 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 20:28:56.073496 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":122,"threads_started":1}
I20260928 20:28:56.074246 14835 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:56.074707 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:56.074859 14835 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:56.084895 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:56.076316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52747 (local address 127.24.186.254:37583)
0928 20:28:56.076353 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:56.076355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:56.076759 (+ 404us) server_negotiation.cc:407] Connection header received
0928 20:28:56.076944 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:56.076945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:56.076956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:56.077015 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:56.077975 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:56.078905 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:56.080049 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:56.080408 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:56.080479 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:56.081968 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:56.081969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:56.081970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:28:56.081978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:56.083627 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:56.084070 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:56.084071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:56.084072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:56.084135 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:28:56.084344 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:28:56.084345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:28:56.084345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:28:56.084474 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:56.084578 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:56.084704 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 20:28:56.084785 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:28:56.086344 14835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f2dbbb085db45659cd1438b4d60d82a: Generated new TSK 0
I20260928 20:28:56.086685 14835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:56.088954 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14757
2026-09-28T20:28:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:56 dist-test-slave-k477 krb5kdc[14856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:56 dist-test-slave-k477 krb5kdc[14856](info): set up 2 sockets
Sep 28 20:28:56 dist-test-slave-k477 krb5kdc[14856](info): commencing operation
krb5kdc: starting...
W20260928 20:28:57.160880 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:57 dist-test-slave-k477 krb5kdc[14856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:57.292951 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38901
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:57.385217 14869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:57.385514 14869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:57.385560 14869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:57.386809 14869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:57.386895 14869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:57.386920 14869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:57.386942 14869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:57.386971 14869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:57.389204 14869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:38901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:57.389580 14869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:57.389858 14869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:57.395460 14875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:57.395496 14874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:57.395586 14869 server_base.cc:1034] running on GCE node
W20260928 20:28:57.395460 14877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:57.397035 14869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:57.397644 14869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:57.398864 14869 hybrid_clock.cc:648] HybridClock initialized: now 1790627337398842 us; error 46 us; skew 500 ppm
Sep 28 20:28:57 dist-test-slave-k477 krb5kdc[14856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627337, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:57.408870 14869 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:57.409425 14869 webserver.cc:460] Webserver started at http://127.24.186.254:32905/ using document root <none> and password file <none>
I20260928 20:28:57.409633 14869 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:57.409677 14869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:57.409772 14869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:57.410838 14869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "fc423476e0ed4510a65666d90ef0f499"
format_stamp: "Formatted at 2026-09-28 20:28:57 on dist-test-slave-k477"
server_key: "828e104bcd6e898390670d79d3409714"
server_key_iv: "fa1b32b8abb860cf90c941dffa26fef2"
server_key_version: "encryptionkey@0"
I20260928 20:28:57.411140 14869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "fc423476e0ed4510a65666d90ef0f499"
format_stamp: "Formatted at 2026-09-28 20:28:57 on dist-test-slave-k477"
server_key: "828e104bcd6e898390670d79d3409714"
server_key_iv: "fa1b32b8abb860cf90c941dffa26fef2"
server_key_version: "encryptionkey@0"
I20260928 20:28:57.412560 14869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 20:28:57.413450 14884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:57.413919 14869 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:57.414063 14869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "fc423476e0ed4510a65666d90ef0f499"
format_stamp: "Formatted at 2026-09-28 20:28:57 on dist-test-slave-k477"
server_key: "828e104bcd6e898390670d79d3409714"
server_key_iv: "fa1b32b8abb860cf90c941dffa26fef2"
server_key_version: "encryptionkey@0"
I20260928 20:28:57.414165 14869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:57.442080 14869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:57.444900 14869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:57.445289 14869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:57.449884 14869 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:38901
I20260928 20:28:57.449919 14936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:38901 every 8 connection(s)
I20260928 20:28:57.450297 14869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:57.451330 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:57.453755 14937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: Bootstrap starting.
I20260928 20:28:57.454344 14937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:57.454581 14937 log.cc:824] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:57.455415 14937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: No bootstrap required, opened a new log
I20260928 20:28:57.457093 14937 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } }
I20260928 20:28:57.457254 14937 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:57.457278 14937 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc423476e0ed4510a65666d90ef0f499, State: Initialized, Role: FOLLOWER
I20260928 20:28:57.457449 14937 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } }
I20260928 20:28:57.457546 14937 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:57.457571 14937 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:57.457602 14937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:57.458487 14937 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } }
I20260928 20:28:57.458587 14937 leader_election.cc:302] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc423476e0ed4510a65666d90ef0f499; no voters:
I20260928 20:28:57.458719 14937 leader_election.cc:288] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:57.458798 14940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:57.458976 14940 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [term 1 LEADER]: Becoming Leader. State: Replica: fc423476e0ed4510a65666d90ef0f499, State: Running, Role: LEADER
I20260928 20:28:57.459010 14937 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:57.459096 14940 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } }
I20260928 20:28:57.459342 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14869
I20260928 20:28:57.459462 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:57.459643 14941 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc423476e0ed4510a65666d90ef0f499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } } }
I20260928 20:28:57.459692 25323 external_mini_cluster.cc:1428] Setting key a8a43a61e744a3a9ba4d2753f96abd3e
I20260928 20:28:57.459726 14941 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:57.459991 14942 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc423476e0ed4510a65666d90ef0f499. Latest consensus state: current_term: 1 leader_uuid: "fc423476e0ed4510a65666d90ef0f499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc423476e0ed4510a65666d90ef0f499" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 38901 } } }
I20260928 20:28:57.460140 14942 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499 [sys.catalog]: This master's current role is: LEADER
W20260928 20:28:57.460889 14957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:28:57.460944 14957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:28:57.460992 14949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:57.461151 14949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:57.462785 14949 catalog_manager.cc:1347] Generated new cluster ID: 9dc5a0b55af347c5bf433ce1e2352c49
I20260928 20:28:57.462842 14949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:57.465639 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:57.460410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60301 (local address 127.24.186.254:38901)
0928 20:28:57.460681 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:57.460690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:57.460724 (+ 34us) server_negotiation.cc:407] Connection header received
0928 20:28:57.460849 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:57.460856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:57.460917 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:57.461120 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:57.461326 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:57.462478 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:57.463453 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:57.463971 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:57.464103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:57.464836 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:57.464840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:57.464841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:57.465043 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:57.465078 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:57.465090 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:57.465168 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:57.465350 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 20:28:57.465427 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":52,"threads_started":1}
I20260928 20:28:57.466536 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:57.476140 14958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:57.472313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52551 (local address 127.24.186.254:38901)
0928 20:28:57.472420 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:57.472423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:57.472771 (+ 348us) server_negotiation.cc:407] Connection header received
0928 20:28:57.472850 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:57.472851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:57.472865 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:57.472912 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:57.473099 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:57.473635 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:57.474548 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:28:57.474779 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:28:57.474864 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:28:57.475848 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:57.475849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:57.475849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:57.475864 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:57.475878 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:57.475882 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:57.475946 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:57.476009 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 20:28:57.476044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 20:28:57.489043 14949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:57.489807 14949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:57.498687 14949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc423476e0ed4510a65666d90ef0f499: Generated new TSK 0
I20260928 20:28:57.498907 14949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:57.502178 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14869
2026-09-28T20:28:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:57 dist-test-slave-k477 krb5kdc[14968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:57 dist-test-slave-k477 krb5kdc[14968](info): set up 2 sockets
Sep 28 20:28:57 dist-test-slave-k477 krb5kdc[14968](info): commencing operation
krb5kdc: starting...
W20260928 20:28:58.576762 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:28:58 dist-test-slave-k477 krb5kdc[14968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:28:58.702947 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41109
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:45629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:28:58.794312 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:28:58.794600 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:28:58.794644 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:28:58.795849 14981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:28:58.795928 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:28:58.795953 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:28:58.795981 14981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:28:58.796006 14981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:28:58.797526 14981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:45629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:41109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:28:58.797837 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:28:58.798091 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:28:58.803772 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:58.803797 14981 server_base.cc:1034] running on GCE node
W20260928 20:28:58.803722 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:28:58.803742 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:28:58.804512 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:28:58.804855 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:28:58.806093 14981 hybrid_clock.cc:648] HybridClock initialized: now 1790627338806074 us; error 37 us; skew 500 ppm
Sep 28 20:28:58 dist-test-slave-k477 krb5kdc[14968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627338, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:28:58.810046 14981 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:28:58.810648 14981 webserver.cc:460] Webserver started at http://127.24.186.254:43769/ using document root <none> and password file <none>
I20260928 20:28:58.810896 14981 fs_manager.cc:359] Metadata directory not provided
I20260928 20:28:58.810968 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:28:58.811126 14981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:28:58.812397 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "c43ac33eeb94455885a7ed15fa3f3a8a"
format_stamp: "Formatted at 2026-09-28 20:28:58 on dist-test-slave-k477"
server_key: "61928e2dd210d5954ed009073fe1ccbb"
server_key_iv: "44d0807230aa51ccc547d72e45cf4cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:58.812798 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "c43ac33eeb94455885a7ed15fa3f3a8a"
format_stamp: "Formatted at 2026-09-28 20:28:58 on dist-test-slave-k477"
server_key: "61928e2dd210d5954ed009073fe1ccbb"
server_key_iv: "44d0807230aa51ccc547d72e45cf4cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:58.815236 14981 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 20:28:58.816342 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:28:58.816635 14981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:28:58.816761 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "c43ac33eeb94455885a7ed15fa3f3a8a"
format_stamp: "Formatted at 2026-09-28 20:28:58 on dist-test-slave-k477"
server_key: "61928e2dd210d5954ed009073fe1ccbb"
server_key_iv: "44d0807230aa51ccc547d72e45cf4cff"
server_key_version: "encryptionkey@0"
I20260928 20:28:58.816848 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:28:58.820611 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:28:58.822991 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:28:58.823199 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:28:58.828711 14981 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:41109
I20260928 20:28:58.828737 15048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:41109 every 8 connection(s)
I20260928 20:28:58.829183 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:28:58.829873 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 14981
I20260928 20:28:58.830025 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:28:58.830298 25323 external_mini_cluster.cc:1428] Setting key 4bb8a407f83affbf64fa232d15cbe691
I20260928 20:28:58.830557 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:28:58.833648 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:58.831157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35267 (local address 127.24.186.254:41109)
0928 20:28:58.831676 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:58.831686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:58.831725 (+ 39us) server_negotiation.cc:407] Connection header received
0928 20:28:58.832277 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:58.832285 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:58.832313 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:58.832440 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:58.833152 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:58.833155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:58.833155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:58.833159 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:58.833197 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:58.833208 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:58.833263 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:58.833354 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 20:28:58.833387 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
I20260928 20:28:58.834364 15049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a: Bootstrap starting.
I20260928 20:28:58.834895 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:28:58.835192 15049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a: Neither blocks nor log segments found. Creating new log.
I20260928 20:28:58.835538 15049 log.cc:824] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a: Log is configured to *not* fsync() on all Append() calls
I20260928 20:28:58.836532 15049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a: No bootstrap required, opened a new log
I20260928 20:28:58.838513 15049 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } }
I20260928 20:28:58.838709 15049 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 20:28:58.838749 15049 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43ac33eeb94455885a7ed15fa3f3a8a, State: Initialized, Role: FOLLOWER
I20260928 20:28:58.838914 15049 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } }
I20260928 20:28:58.839028 15049 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 20:28:58.839076 15049 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 20:28:58.839130 15049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:28:58.839792 15049 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } }
I20260928 20:28:58.839975 15049 leader_election.cc:302] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43ac33eeb94455885a7ed15fa3f3a8a; no voters:
I20260928 20:28:58.840204 15049 leader_election.cc:288] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 20:28:58.840332 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:28:58.836741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48577 (local address 127.24.186.254:41109)
0928 20:28:58.839072 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0928 20:28:58.839075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:28:58.839088 (+ 13us) server_negotiation.cc:407] Connection header received
0928 20:28:58.839128 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:28:58.839130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:28:58.839135 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:28:58.839208 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:28:58.839937 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:28:58.839938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:28:58.839939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 20:28:58.839943 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:28:58.839958 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 20:28:58.839963 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:28:58.840024 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:28:58.840191 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 20:28:58.840205 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2284}
I20260928 20:28:58.840655 15049 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:28:58.840744 15059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:28:58.841099 15059 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [term 1 LEADER]: Becoming Leader. State: Replica: c43ac33eeb94455885a7ed15fa3f3a8a, State: Running, Role: LEADER
I20260928 20:28:58.841301 15059 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } }
I20260928 20:28:58.841987 15060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } } }
I20260928 20:28:58.842123 15060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:58.841998 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43ac33eeb94455885a7ed15fa3f3a8a. Latest consensus state: current_term: 1 leader_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43ac33eeb94455885a7ed15fa3f3a8a" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 41109 } } }
I20260928 20:28:58.842350 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a [sys.catalog]: This master's current role is: LEADER
I20260928 20:28:58.842550 15068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:28:58.843086 15068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:28:58.844794 15068 catalog_manager.cc:1347] Generated new cluster ID: 8d1b0ff33cc54e49a7f1ced3f637cc34
I20260928 20:28:58.844847 15068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:28:58.859812 15068 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:28:58.860014 15068 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:28:58.866582 15068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43ac33eeb94455885a7ed15fa3f3a8a: Generated new TSK 0
I20260928 20:28:58.866855 15068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:28:58.870852 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 14981
2026-09-28T20:28:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1369 ms)
[----------] 18 tests from AuthTokenIssuingTest (24742 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:28:58 dist-test-slave-k477 krb5kdc[15080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:28:58 dist-test-slave-k477 krb5kdc[15080](info): set up 2 sockets
Sep 28 20:28:58 dist-test-slave-k477 krb5kdc[15080](info): commencing operation
krb5kdc: starting...
W20260928 20:28:59.936597 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:29:00.071245 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:42891
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:29:00.161245 15093 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:00.161576 15093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:00.161613 15093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:00.162602 15093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:00.164569 15093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:00.164712 15093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:00.164740 15093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:00.164760 15093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:00.164789 15093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:00.166740 15093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:00.167119 15093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:00.167390 15093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:29:00.173156 15098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:00.173256 15101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:00.173156 15099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:00.174594 15093 server_base.cc:1034] running on GCE node
I20260928 20:29:00.175014 15093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:00.175362 15093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:00.176630 15093 hybrid_clock.cc:648] HybridClock initialized: now 1790627340176598 us; error 47 us; skew 500 ppm
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:00.180608 15093 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:29:00.181798 15093 webserver.cc:460] Webserver started at http://127.24.186.254:37129/ using document root <none> and password file <none>
I20260928 20:29:00.182183 15093 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:00.182274 15093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:00.182483 15093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:00.184295 15093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "397a77c2d5bf478c96dfa8e432333119"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "61c34919d9cf9fd7197ab95e87aba4e6"
server_key_iv: "4879ddf710d2da7f8c93f7f2e106730d"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.184942 15093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "397a77c2d5bf478c96dfa8e432333119"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "61c34919d9cf9fd7197ab95e87aba4e6"
server_key_iv: "4879ddf710d2da7f8c93f7f2e106730d"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.186936 15093 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 20:29:00.188292 15108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:00.188635 15093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:29:00.188783 15093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "397a77c2d5bf478c96dfa8e432333119"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "61c34919d9cf9fd7197ab95e87aba4e6"
server_key_iv: "4879ddf710d2da7f8c93f7f2e106730d"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.188889 15093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:00.211592 15093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:00.214572 15093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:00.214792 15093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:00.220512 15093 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:42891
I20260928 20:29:00.220548 15160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:42891 every 8 connection(s)
I20260928 20:29:00.221045 15093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:29:00.222568 15161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:00.224752 15161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.227676 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15093
I20260928 20:29:00.227864 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:29:00.228130 25323 external_mini_cluster.cc:1428] Setting key 4be96333f3e5b5fd33509374ad818ecc
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.24.186.253@KRBTEST.COM; defaulting to no policy
I20260928 20:29:00.240027 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.225852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.24.186.254:42891)
0928 20:29:00.226055 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.226066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.226249 (+ 183us) server_negotiation.cc:407] Connection header received
0928 20:29:00.226998 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.227001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.227021 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.227120 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.228133 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.229513 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.231364 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.232106 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.232266 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.235605 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.235613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.235613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.235621 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.237625 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.238127 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.238130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.238130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.238217 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.238666 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.238667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.238667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.238907 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.238979 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.239549 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.239738 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260928 20:29:00.240041 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.225674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42891 (local address 127.0.0.1:59162)
0928 20:29:00.226006 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.226058 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.226780 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.227194 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.227199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.227546 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.227787 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.227791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.229723 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.229726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.231165 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.231171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.232334 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.232339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.232453 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.233325 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.233337 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.235438 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.237800 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.237808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.237810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.238025 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.238492 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.238493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.238493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.238601 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.239047 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.239053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.239119 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.239534 (+ 415us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.239824 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":105,"thread_start_us":226,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.253@KRBTEST.COM" created.
I20260928 20:29:00.240898 15161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.241279 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.241152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59658)
0928 20:29:00.241204 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.241252 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 20:29:00.241405 15111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
I20260928 20:29:00.242233 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.242023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59660)
0928 20:29:00.242177 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.242205 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260928 20:29:00.242383 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:45309: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
I20260928 20:29:00.260869 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:45309
--webserver_interface=127.24.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:29:00.291711 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } attempt: 1
I20260928 20:29:00.292265 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.292121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59662)
0928 20:29:00.292190 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.292236 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 20:29:00.292680 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.292595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59676)
0928 20:29:00.292638 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.292663 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:29:00.292811 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:45309: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
W20260928 20:29:00.354550 15168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:00.354799 15168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:00.354817 15168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:00.354841 15168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:00.356055 15168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:00.356099 15168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:00.356113 15168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:00.356127 15168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:00.356138 15168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:00.357713 15168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:00.357941 15168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:00.358136 15168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 20:29:00.360983 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } attempt: 2
I20260928 20:29:00.361467 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.361323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59684)
0928 20:29:00.361383 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.361425 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 20:29:00.361829 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.361744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59700)
0928 20:29:00.361787 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.361813 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 20:29:00.361953 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:45309: Network error: Client connection negotiation failed: client connection to 127.24.186.253:45309: connect: Connection refused (error 111)
W20260928 20:29:00.363286 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:00.363286 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:00.363420 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:00.363794 15168 server_base.cc:1034] running on GCE node
I20260928 20:29:00.364064 15168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:00.364444 15168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:00.365658 15168 hybrid_clock.cc:648] HybridClock initialized: now 1790627340365625 us; error 43 us; skew 500 ppm
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:00.368701 15168 init.cc:377] Logged in from keytab as kudu/127.24.186.253@KRBTEST.COM (short username kudu)
I20260928 20:29:00.369234 15168 webserver.cc:460] Webserver started at http://127.24.186.253:44019/ using document root <none> and password file <none>
I20260928 20:29:00.369488 15168 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:00.369558 15168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:00.369721 15168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:00.370846 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data/instance:
uuid: "35cc63d65ca84fffab8098026a70dd50"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "543cc9d754697242a4056d612656ad5a"
server_key_iv: "7b87c7fe4c3eca0a9f22b4283b3c3e6e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.371227 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal/instance:
uuid: "35cc63d65ca84fffab8098026a70dd50"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "543cc9d754697242a4056d612656ad5a"
server_key_iv: "7b87c7fe4c3eca0a9f22b4283b3c3e6e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.372761 15168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 20:29:00.373862 15183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:00.374269 15168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:29:00.374418 15168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "35cc63d65ca84fffab8098026a70dd50"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "543cc9d754697242a4056d612656ad5a"
server_key_iv: "7b87c7fe4c3eca0a9f22b4283b3c3e6e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.374523 15168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:00.394297 15168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:00.397477 15168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:00.397742 15168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:00.402879 15168 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.253:45309
I20260928 20:29:00.402915 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.253:45309 every 8 connection(s)
I20260928 20:29:00.403301 15168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
I20260928 20:29:00.404277 15236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:00.405874 15236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.408879 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15168
I20260928 20:29:00.409058 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-1/wal/instance
I20260928 20:29:00.409529 25323 external_mini_cluster.cc:1428] Setting key 7e16e3fd7e4358688e2f474b0c7c8770
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:00.416769 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.406653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.24.186.254:42891)
0928 20:29:00.406730 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.406732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.407079 (+ 347us) server_negotiation.cc:407] Connection header received
0928 20:29:00.407582 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.407584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.407595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.407659 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.408532 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.409047 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.410804 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.411067 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.411390 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.413790 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.413793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.413793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.413804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.415262 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.415709 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.415711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.415711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.415774 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.416088 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.416089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.416090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.416219 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.416297 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.416585 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.416671 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 20:29:00.416975 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.406629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42891 (local address 127.0.0.1:59174)
0928 20:29:00.406863 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.406912 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.407419 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.407773 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.407778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.408185 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.408405 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.408409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.409321 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.409322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.410682 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.410686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.411217 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.411218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.411353 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.411986 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.411995 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.413672 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.415373 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.415382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.415385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.415630 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.415885 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.415886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.415887 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.416010 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.416289 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.416295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.416450 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.416565 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.416732 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.252@KRBTEST.COM; defaulting to no policy
I20260928 20:29:00.417711 15236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.252@KRBTEST.COM" created.
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:00.426486 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.418020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59712)
0928 20:29:00.418086 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.418096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.418138 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.418389 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.418390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.418544 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.418593 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.418594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.419501 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.419502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.420078 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.420081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.420599 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.420599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.420666 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.421384 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.421391 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.422952 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.425336 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.425337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.425338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.425514 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.425766 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.425766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.425767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.425818 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.426218 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.426219 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.426262 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.426330 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.426414 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 20:29:00.426656 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.418081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.24.186.253:45309)
0928 20:29:00.418253 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.418258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.418274 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:29:00.418290 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.418292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.418302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.418390 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.418668 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.419372 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.420172 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.420503 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.420601 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.423061 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.423075 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.423076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.423085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.425235 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.425619 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.425623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.425624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.425694 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.425879 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.425880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.425880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.426099 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.426195 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.426399 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.426494 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260928 20:29:00.427433 15236 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.427907 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.427734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41904)
0928 20:29:00.427795 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.427877 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 20:29:00.428081 15185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
I20260928 20:29:00.428663 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.428545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41916)
0928 20:29:00.428599 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.428631 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 20:29:00.428781 15236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
I20260928 20:29:00.440042 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:46721
--webserver_interface=127.24.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:36855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:29:00.453174 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } attempt: 3
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:00.462049 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.453759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59728)
0928 20:29:00.453854 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.453866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.453909 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.454110 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.454111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.454318 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.454392 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.454394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.455101 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.455103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.456020 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.456023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.456463 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.456464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.456547 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.457242 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.457248 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.459141 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.460939 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.460941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.460942 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.461127 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.461334 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.461335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.461335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.461399 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.461700 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.461701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.461749 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.461833 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.461939 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 20:29:00.462049 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.453885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.24.186.253:45309)
0928 20:29:00.453948 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.453952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.453972 (+ 20us) server_negotiation.cc:407] Connection header received
0928 20:29:00.453999 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.454000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.454010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.454076 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.454495 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.455004 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.456136 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.456378 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.456448 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.459270 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.459274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.459275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.459289 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.460826 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.461203 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.461204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.461205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.461265 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.461494 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.461495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.461495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.461620 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.461695 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.461858 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.461940 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 20:29:00.462685 15161 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.463033 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.462933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41932)
0928 20:29:00.462985 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.463014 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:29:00.463352 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.463266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41940)
0928 20:29:00.463310 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.463333 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:29:00.463443 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
I20260928 20:29:00.478018 15236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 1
I20260928 20:29:00.478641 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.478492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41942)
0928 20:29:00.478559 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.478606 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:29:00.479069 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.478925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41944)
0928 20:29:00.479009 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.479045 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 20:29:00.479213 15236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
I20260928 20:29:00.494599 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 1
I20260928 20:29:00.495113 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.494978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41956)
0928 20:29:00.495045 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.495086 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:29:00.495543 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.495400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41970)
0928 20:29:00.495465 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.495498 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 20:29:00.495672 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
W20260928 20:29:00.531404 15243 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:00.531701 15243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:00.531733 15243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:00.531780 15243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:00.533164 15243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:00.533243 15243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:00.533267 15243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:00.533289 15243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:00.533334 15243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:00.534946 15243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:36855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:42891,127.24.186.253:45309,127.24.186.252:46721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:46721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:00.535260 15243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:00.535507 15243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:29:00.540840 15249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:00.540890 15251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:00.540949 15243 server_base.cc:1034] running on GCE node
W20260928 20:29:00.540913 15248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:00.541612 15243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:00.542011 15243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:00.543226 15243 hybrid_clock.cc:648] HybridClock initialized: now 1790627340543207 us; error 117 us; skew 500 ppm
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:00.546129 15243 init.cc:377] Logged in from keytab as kudu/127.24.186.252@KRBTEST.COM (short username kudu)
I20260928 20:29:00.546627 15243 webserver.cc:460] Webserver started at http://127.24.186.252:43011/ using document root <none> and password file <none>
I20260928 20:29:00.546864 15243 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:00.546936 15243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:00.547094 15243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:00.547366 15236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 2
I20260928 20:29:00.547776 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.547632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41986)
0928 20:29:00.547700 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.547740 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 20:29:00.548135 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.548030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41992)
0928 20:29:00.548080 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.548110 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260928 20:29:00.548363 15236 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
I20260928 20:29:00.548535 15243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data/instance:
uuid: "ece073475712455c8d643f8698b93938"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "b48ef1ef04c34f9e63b9f1ad6bde13ce"
server_key_iv: "1a4a4071be7e0e3f0ca0af470179358e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.548906 15243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal/instance:
uuid: "ece073475712455c8d643f8698b93938"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "b48ef1ef04c34f9e63b9f1ad6bde13ce"
server_key_iv: "1a4a4071be7e0e3f0ca0af470179358e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.550387 15243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:29:00.551645 15258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:00.552080 15243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:29:00.552186 15243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "ece073475712455c8d643f8698b93938"
format_stamp: "Formatted at 2026-09-28 20:29:00 on dist-test-slave-k477"
server_key: "b48ef1ef04c34f9e63b9f1ad6bde13ce"
server_key_iv: "1a4a4071be7e0e3f0ca0af470179358e"
server_key_version: "encryptionkey@0"
I20260928 20:29:00.552284 15243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:00.570852 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 2
I20260928 20:29:00.571374 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.571238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:41998)
0928 20:29:00.571298 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.571344 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 20:29:00.571780 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.571698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:42008)
0928 20:29:00.571740 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.571764 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 20:29:00.571884 15161 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:46721: Network error: Client connection negotiation failed: client connection to 127.24.186.252:46721: connect: Connection refused (error 111)
I20260928 20:29:00.582264 15243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:00.584968 15243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:00.585139 15243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:00.589644 15243 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.252:46721
I20260928 20:29:00.589677 15310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.252:46721 every 8 connection(s)
I20260928 20:29:00.590054 15243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
I20260928 20:29:00.591105 15311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:00.593035 15311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.596025 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15243
I20260928 20:29:00.596118 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790627060930553-25323-0/minicluster-data/master-2/wal/instance
I20260928 20:29:00.596313 25323 external_mini_cluster.cc:1428] Setting key 9ea4dbc52ee965b44993db8741f439e4
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:29:00.606132 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.593811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.24.186.254:42891)
0928 20:29:00.593903 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.593905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.594096 (+ 191us) server_negotiation.cc:407] Connection header received
0928 20:29:00.594589 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.594590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.594600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.594662 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.595283 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.595817 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.597269 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.597600 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.597697 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.601126 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.601136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.601137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.601147 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.603296 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.604377 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.604379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.604379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.604448 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.604817 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.604818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.604818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.605237 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.605301 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.605876 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.605999 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 20:29:00.606503 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.593764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:42891 (local address 127.0.0.1:59182)
0928 20:29:00.593964 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.594000 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.594480 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.594709 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.594712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.594981 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.595186 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.595190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.595950 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.595951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.597172 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.597174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.597720 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.597721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.597832 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.598369 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.598379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.600975 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.603921 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.603931 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.603933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.604270 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.604524 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.604525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.604526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.604678 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.605343 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.605362 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.605671 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.605834 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.606049 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260928 20:29:00.607429 15311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.608654 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.596773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.24.186.252:46721)
0928 20:29:00.596974 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.596979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.597022 (+ 43us) server_negotiation.cc:407] Connection header received
0928 20:29:00.597052 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.597054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.597074 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.597149 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.597974 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.599013 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.600712 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.601183 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.601244 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.604592 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.604597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.604598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.604606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.607226 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.607666 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.607669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.607669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.607725 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.608036 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.608037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.608037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.608205 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.608394 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.608448 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.608538 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:00.616640 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.607852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.24.186.253:45309)
0928 20:29:00.607907 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.607909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.607918 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:00.607946 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.607947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.607956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.608007 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.608294 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.608883 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.609834 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.610106 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.610167 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.612518 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.612520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.612520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.612528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.614574 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.615122 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.615124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.615124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.615205 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.615694 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.615695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.615696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.615981 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.616076 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.616396 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.616510 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 20:29:00.616642 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.607758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:45309 (local address 127.0.0.1:59740)
0928 20:29:00.607837 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.607848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.607891 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.608010 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.608011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.608159 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.608202 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.608204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.608962 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.608963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.609747 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.609749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.610163 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.610164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.610225 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.610756 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.610761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.612300 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.614721 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.614723 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.614724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.614978 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.615450 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.615452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.615453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.615566 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.616064 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.616065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.616129 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.616340 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.616510 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 20:29:00.617277 15311 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:00.619575 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.609515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.24.186.254:42891)
0928 20:29:00.609558 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.609560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.609569 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:00.609612 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.609613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.609623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.609684 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.610316 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.610911 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.611955 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.612251 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.612333 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.616252 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.616254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.616254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.616263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.618328 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.618721 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.618722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.618723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.618785 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.619003 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.619003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.619004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.619122 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.619327 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.619365 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.619444 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:00.626746 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.617676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:42022)
0928 20:29:00.617731 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.617743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.617783 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.618012 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.618014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.618190 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.618324 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.618328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.619118 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.619119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.619616 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.619617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.620153 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.620154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.620411 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.621127 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.621134 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.622993 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.625399 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.625400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.625401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.625693 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.625964 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.625965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.625965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.626055 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.626323 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.626324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.626377 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.626467 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.626595 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 20:29:00.626746 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.617772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.24.186.252:46721)
0928 20:29:00.617834 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.617836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.617847 (+ 11us) server_negotiation.cc:407] Connection header received
0928 20:29:00.617875 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.617876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.617893 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.617977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.618458 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.618987 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.619734 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.620075 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.620142 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.623122 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.623124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.623124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.623134 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.625277 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.625816 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.625818 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.625818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.625888 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.626126 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.626126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.626127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.626246 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.626296 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.626519 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.626648 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":34}
I20260928 20:29:00.628607 15311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: Bootstrap starting.
I20260928 20:29:00.629521 15311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:00.629839 15311 log.cc:824] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:00.629895 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.620709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.24.186.253:45309)
0928 20:29:00.620807 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.620810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.620819 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:00.620852 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.620853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.620865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.621002 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.621847 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.622645 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.623571 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.623777 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.623908 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.626531 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.626533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.626533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.626541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.628647 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.629052 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.629053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.629053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.629121 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.629332 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.629333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.629334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.629462 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.629647 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.629693 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.629780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 20:29:00.630801 15311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: No bootstrap required, opened a new log
I20260928 20:29:00.632701 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:29:00.632866 15311 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.633057 15311 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece073475712455c8d643f8698b93938, State: Initialized, Role: FOLLOWER
I20260928 20:29:00.633162 15311 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.633481 15311 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:00.634063 15317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.634256 15317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:29:00.635535 15328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:00.635635 15328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:29:00.634554 15135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "397a77c2d5bf478c96dfa8e432333119"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59190
I20260928 20:29:00.635672 15243 master_runner.cc:186] Error getting master registration for 127.24.186.254:42891: OK, instance_id { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" instance_seqno: 1790627340218680 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:29:00.638157 15243 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:29:00.639533 15236 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 3
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:29:00.647019 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.640024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:42026)
0928 20:29:00.640081 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.640095 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.640156 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.640317 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.640318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.640491 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.640537 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.640538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.641178 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.641178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.641684 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.641685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.642058 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.642058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.642118 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.642634 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.642640 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.644202 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.645881 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.645883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.645883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.646070 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.646296 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.646296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.646297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.646364 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.646676 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.646677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.646725 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.646810 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.646918 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 20:29:00.647019 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.640087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42026 (local address 127.24.186.252:46721)
0928 20:29:00.640140 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.640142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.640151 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:00.640222 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.640223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.640233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.640297 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.640620 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.641091 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.641772 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.641982 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.642061 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.644303 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.644305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.644305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.644313 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.645770 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.646166 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.646168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.646168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.646228 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.646430 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.646431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.646432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.646596 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.646672 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.646834 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.646918 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 20:29:00.648741 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: Bootstrap starting.
I20260928 20:29:00.649461 15236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:00.649757 15236 log.cc:824] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:00.650699 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: No bootstrap required, opened a new log
I20260928 20:29:00.652308 15236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.652458 15236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cc63d65ca84fffab8098026a70dd50, State: Initialized, Role: FOLLOWER
I20260928 20:29:00.652575 15236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.653002 15236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:00.652959 15331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.653108 15331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:29:00.655033 15342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:00.655098 15342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:29:00.655474 15168 master_runner.cc:186] Error getting master registration for 127.24.186.254:42891: OK, instance_id { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" instance_seqno: 1790627340218680 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:29:00.658586 15168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:29:00.671067 15161 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } attempt: 3
Sep 28 20:29:00 dist-test-slave-k477 krb5kdc[15080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627340, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:29:00.681121 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.671449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:46721 (local address 127.0.0.1:42042)
0928 20:29:00.671516 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:00.672337 (+ 821us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:00.672471 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:00.672667 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:00.672668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:00.673042 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:00.673118 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.673123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.674698 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.674699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.675240 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:00.675242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.675791 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.675792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:00.675875 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.676467 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:00.676472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:00.678148 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:00.679949 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.679951 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.679951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.680198 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.680393 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:00.680394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:00.680394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:00.680475 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:00.680774 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:00.680776 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:00.680830 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 20:29:00.680901 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 20:29:00.681012 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 20:29:00.681121 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.671521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.24.186.252:46721)
0928 20:29:00.672298 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.672302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.672431 (+ 129us) server_negotiation.cc:407] Connection header received
0928 20:29:00.672556 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.672557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.672571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.672657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.673388 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.674560 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.675349 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.675665 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.675727 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.678341 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.678343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.678343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.678352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.679783 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.680282 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.680283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.680284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.680340 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.680548 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.680549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.680549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.680713 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.680774 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.680943 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.681016 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726}
I20260928 20:29:00.683633 15161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: Bootstrap starting.
I20260928 20:29:00.684412 15161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:00.684703 15161 log.cc:824] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:00.685600 15161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: No bootstrap required, opened a new log
I20260928 20:29:00.687347 15161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.687505 15161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 397a77c2d5bf478c96dfa8e432333119, State: Initialized, Role: FOLLOWER
I20260928 20:29:00.687582 15161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.687943 15161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:00.688047 15135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 20:29:00.688107 15135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:00.689081 15135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
I20260928 20:29:00.689364 15135 leader_election.cc:288] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [CANDIDATE]: Term 1 election: Requested vote from peers 35cc63d65ca84fffab8098026a70dd50 (127.24.186.253:45309), ece073475712455c8d643f8698b93938 (127.24.186.252:46721)
I20260928 20:29:00.689752 15210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "397a77c2d5bf478c96dfa8e432333119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35cc63d65ca84fffab8098026a70dd50"
I20260928 20:29:00.689755 15285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "397a77c2d5bf478c96dfa8e432333119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ece073475712455c8d643f8698b93938"
I20260928 20:29:00.689883 15285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:00.689877 15210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:00.690794 15285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397a77c2d5bf478c96dfa8e432333119 in term 1.
I20260928 20:29:00.690794 15210 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 397a77c2d5bf478c96dfa8e432333119 in term 1.
I20260928 20:29:00.691923 15111 leader_election.cc:302] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cc63d65ca84fffab8098026a70dd50, 397a77c2d5bf478c96dfa8e432333119; no voters:
I20260928 20:29:00.693624 15345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.693835 15345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:00.693943 15345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:29:00.694028 15345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [term 1 LEADER]: Becoming Leader. State: Replica: 397a77c2d5bf478c96dfa8e432333119, State: Running, Role: LEADER
I20260928 20:29:00.694162 15345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } }
W20260928 20:29:00.694703 15361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:00.694756 15361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:29:00.694978 15345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397a77c2d5bf478c96dfa8e432333119. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.695116 15345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: This master's current role is: LEADER
I20260928 20:29:00.696528 15364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:29:00.697033 15364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:29:00.704249 15285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [term 1 FOLLOWER]: Refusing update from remote peer 397a77c2d5bf478c96dfa8e432333119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:29:00.704671 15210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [term 1 FOLLOWER]: Refusing update from remote peer 397a77c2d5bf478c96dfa8e432333119: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:29:00.705029 15355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:29:00.705462 15355 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:29:00.707903 15317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397a77c2d5bf478c96dfa8e432333119. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.708010 15317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:00.709110 15331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 397a77c2d5bf478c96dfa8e432333119. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.709223 15331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:00.710122 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.695537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.24.186.252:46721)
0928 20:29:00.695585 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.695588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.700060 (+ 4472us) server_negotiation.cc:407] Connection header received
0928 20:29:00.700178 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.700179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.700193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.700226 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.701169 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.701946 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.704058 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.704319 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.704364 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.706541 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.706542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.706543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.706552 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.708518 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.709051 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.709053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.709053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.709114 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.709433 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.709434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.709434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.709571 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.709672 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.709950 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.710031 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:00.715878 15114 tablet.cc:2369] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 20:29:00.717765 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.717908 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: This master's current role is: LEADER
I20260928 20:29:00.718346 15331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.718449 15331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:00.718657 15366 mvcc.cc:205] Tried to move back new op lower bound from 7334409587518214144 to 7334409587481505792. Current Snapshot: MvccSnapshot[applied={T|T < 7334409587518214144}]
I20260928 20:29:00.719151 15371 mvcc.cc:205] Tried to move back new op lower bound from 7334409587518214144 to 7334409587481505792. Current Snapshot: MvccSnapshot[applied={T|T < 7334409587518214144}]
I20260928 20:29:00.719707 15240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.700422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.24.186.253:45309)
0928 20:29:00.700466 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.700468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.710992 (+ 10524us) server_negotiation.cc:407] Connection header received
0928 20:29:00.711091 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.711092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.711105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.711137 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.711866 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.712688 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.714131 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.714405 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.714448 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.716113 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.716114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.716115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.716122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.717915 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.718642 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.718644 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.718644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.718706 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.718964 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.718965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.718966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.719121 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.719211 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.719514 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.719604 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:29:00.719864 15345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.719969 15345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119 [sys.catalog]: This master's current role is: LEADER
I20260928 20:29:00.720410 15317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "397a77c2d5bf478c96dfa8e432333119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "397a77c2d5bf478c96dfa8e432333119" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 42891 } } peers { permanent_uuid: "35cc63d65ca84fffab8098026a70dd50" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 45309 } } peers { permanent_uuid: "ece073475712455c8d643f8698b93938" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 46721 } } }
I20260928 20:29:00.720506 15317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:00.720779 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:00.700080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.24.186.254:42891)
0928 20:29:00.700124 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.700127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.707989 (+ 7862us) server_negotiation.cc:407] Connection header received
0928 20:29:00.708097 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.708098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.708110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.708141 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.708788 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.709539 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.710587 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:00.710881 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:00.710922 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:00.715909 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:00.715911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:00.715911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:00.715920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:00.718008 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.719133 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.719135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.719135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.719198 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:00.719395 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:00.719396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:00.719396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:00.719521 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:00.719608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:00.719832 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 20:29:00.719915 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:29:00.721372 15369 mvcc.cc:205] Tried to move back new op lower bound from 7334409587518214144 to 7334409587481505792. Current Snapshot: MvccSnapshot[applied={T|T < 7334409587518214144}]
I20260928 20:29:00.721594 15364 catalog_manager.cc:1347] Generated new cluster ID: 7b479192975647bf96334a5a6d0ac60f
I20260928 20:29:00.721643 15364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:29:00.734416 15364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:29:00.735138 15364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:29:00.740891 15364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 397a77c2d5bf478c96dfa8e432333119: Generated new TSK 0
I20260928 20:29:00.741108 15364 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 20:29:00.765998 15164 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.764239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.24.186.254:42891)
0928 20:29:00.764289 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.764292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.765237 (+ 945us) server_negotiation.cc:407] Connection header received
0928 20:29:00.765353 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.765354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.765367 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.765399 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.765905 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:29:00.766810 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.764620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.24.186.253:45309)
0928 20:29:00.764662 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.764663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.766265 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 20:29:00.766365 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.766366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.766378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.766407 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.766739 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59772: BlockingRecv error: recv got EOF from 127.0.0.1:59772 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 20:29:00.768095 15315 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.764951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.24.186.252:46721)
0928 20:29:00.764998 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.765000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.767066 (+ 2066us) server_negotiation.cc:407] Connection header received
0928 20:29:00.767174 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.767175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.767187 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.767217 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.767994 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 20:29:00.773849 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.772725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.24.186.253:45309)
0928 20:29:00.772775 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.772777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.773157 (+ 380us) server_negotiation.cc:407] Connection header received
0928 20:29:00.773295 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.773296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.773317 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.773347 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.773781 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59784: BlockingRecv error: recv got EOF from 127.0.0.1:59784 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:29:00.777395 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.776432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59788 (local address 127.24.186.253:45309)
0928 20:29:00.776476 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.776478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.776740 (+ 262us) server_negotiation.cc:407] Connection header received
0928 20:29:00.776844 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.776845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.776856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.776887 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.777329 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59788: BlockingRecv error: recv got EOF from 127.0.0.1:59788 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:00.781647 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.780723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.24.186.253:45309)
0928 20:29:00.780767 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.780769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.781025 (+ 256us) server_negotiation.cc:407] Connection header received
0928 20:29:00.781129 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.781130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.781143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.781174 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.781577 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59794: BlockingRecv error: recv got EOF from 127.0.0.1:59794 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 20:29:00.787981 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.786974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.24.186.253:45309)
0928 20:29:00.787016 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.787019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.787260 (+ 241us) server_negotiation.cc:407] Connection header received
0928 20:29:00.787366 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.787367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.787379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.787410 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.787920 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59800: BlockingRecv error: recv got EOF from 127.0.0.1:59800 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:00.798758 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.797766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.24.186.253:45309)
0928 20:29:00.797816 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.797819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.798108 (+ 289us) server_negotiation.cc:407] Connection header received
0928 20:29:00.798244 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.798245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.798257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.798288 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.798694 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59806: BlockingRecv error: recv got EOF from 127.0.0.1:59806 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:00.817371 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.816335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.24.186.253:45309)
0928 20:29:00.816386 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.816388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.816718 (+ 330us) server_negotiation.cc:407] Connection header received
0928 20:29:00.816823 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.816824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.816836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.816865 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.817309 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59820: BlockingRecv error: recv got EOF from 127.0.0.1:59820 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 20:29:00.852551 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.850866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.24.186.253:45309)
0928 20:29:00.850921 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.850923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.851388 (+ 465us) server_negotiation.cc:407] Connection header received
0928 20:29:00.851503 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.851504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.851517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.851548 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.852487 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59822: BlockingRecv error: recv got EOF from 127.0.0.1:59822 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 20:29:00.919467 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:00.918395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.24.186.253:45309)
0928 20:29:00.918464 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:00.918467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:00.918715 (+ 248us) server_negotiation.cc:407] Connection header received
0928 20:29:00.918756 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:00.918757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:00.918770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:00.918842 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:00.919407 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59834: BlockingRecv error: recv got EOF from 127.0.0.1:59834 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 20:29:01.049760 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:01.048910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59842 (local address 127.24.186.253:45309)
0928 20:29:01.049022 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.049025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.049134 (+ 109us) server_negotiation.cc:407] Connection header received
0928 20:29:01.049243 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.049244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.049257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.049335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:01.049696 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59842: BlockingRecv error: recv got EOF from 127.0.0.1:59842 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 20:29:01.307721 15240 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:01.307165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.24.186.253:45309)
0928 20:29:01.307223 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.307225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.307235 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:29:01.307277 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.307277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.307288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.307332 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:01.307641 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59856: BlockingRecv error: recv got EOF from 127.0.0.1:59856 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 20:29:01.636149 15328 catalog_manager.cc:1259] Loaded cluster ID: 7b479192975647bf96334a5a6d0ac60f
I20260928 20:29:01.636231 15328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: loading cluster ID for follower catalog manager: success
I20260928 20:29:01.637053 15328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: acquiring CA information for follower catalog manager: success
I20260928 20:29:01.637367 15328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ece073475712455c8d643f8698b93938: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:29:01.655570 15342 catalog_manager.cc:1259] Loaded cluster ID: 7b479192975647bf96334a5a6d0ac60f
I20260928 20:29:01.655646 15342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: loading cluster ID for follower catalog manager: success
I20260928 20:29:01.656584 15342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: acquiring CA information for follower catalog manager: success
I20260928 20:29:01.656893 15342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35cc63d65ca84fffab8098026a70dd50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:29:01.824386 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:01.821431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.24.186.253:45309)
0928 20:29:01.821603 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.821605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.821633 (+ 28us) server_negotiation.cc:407] Connection header received
0928 20:29:01.821713 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.821714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.821763 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.821841 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:01.822166 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.822679 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.823541 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.823746 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.823825 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:01.824030 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:01.824146 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:01.824206 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:01.824242 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 20:29:01.824260 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260928 20:29:01.829276 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:01.826438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.24.186.252:46721)
0928 20:29:01.826581 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.826583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.826592 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:01.826636 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.826637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.826670 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.826740 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:01.826843 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.827320 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.828148 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.828339 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.828403 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:01.828678 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:01.828853 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:01.828968 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:01.829107 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 20:29:01.829163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260928 20:29:01.834414 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:01.831371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.24.186.252:46721)
0928 20:29:01.831413 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.831414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.831485 (+ 71us) server_negotiation.cc:407] Connection header received
0928 20:29:01.831564 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.831565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.831575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.831622 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:01.831772 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.832551 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833553 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.833859 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833944 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:01.834076 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:01.834111 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:01.834168 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:01.834295 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 20:29:01.834318 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:29:01.834441 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:01.831549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.24.186.253:45309)
0928 20:29:01.831594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.831596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.831642 (+ 46us) server_negotiation.cc:407] Connection header received
0928 20:29:01.831705 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.831706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.831718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.831772 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:01.831931 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.832447 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833562 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.833860 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833937 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:01.834152 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:01.834187 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:01.834217 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:01.834324 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 20:29:01.834348 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 20:29:01.834602 15453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:01.831342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.24.186.254:42891)
0928 20:29:01.831748 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:01.831750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:01.831758 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:29:01.831828 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:01.831829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:01.831885 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:01.831966 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:01.832078 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.832775 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833609 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:01.833937 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:01.833993 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:01.834237 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:01.834352 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:01.834412 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:01.834473 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 20:29:01.834499 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":53,"threads_started":1}
I20260928 20:29:01.836549 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15093
I20260928 20:29:01.842970 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15168
I20260928 20:29:01.848182 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15243
2026-09-28T20:29:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2979 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 20:29:01 dist-test-slave-k477 krb5kdc[15457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 20:29:01 dist-test-slave-k477 krb5kdc[15457](info): set up 2 sockets
Sep 28 20:29:01 dist-test-slave-k477 krb5kdc[15457](info): commencing operation
krb5kdc: starting...
W20260928 20:29:02.928755 25323 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/test-user.keytab.
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T20:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T20:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.24.186.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
Entry for principal kudu/127.24.186.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab.
I20260928 20:29:03.054860 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35649
--webserver_interface=127.24.186.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 20:29:03.139992 15470 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:03.140413 15470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:03.140493 15470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:03.140548 15470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:03.142037 15470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:03.142130 15470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:03.142159 15470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:03.142189 15470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:03.142218 15470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:03.143945 15470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.254.keytab
--principal=kudu/127.24.186.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.254:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.24.186.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:03.144258 15470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:03.144498 15470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:29:03.149408 15475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:03.149454 15476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:03.149502 15478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:03.149544 15470 server_base.cc:1034] running on GCE node
I20260928 20:29:03.149869 15470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:03.150141 15470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:03.151305 15470 hybrid_clock.cc:648] HybridClock initialized: now 1790627343151284 us; error 41 us; skew 500 ppm
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:03.159584 15470 init.cc:377] Logged in from keytab as kudu/127.24.186.254@KRBTEST.COM (short username kudu)
I20260928 20:29:03.160167 15470 webserver.cc:460] Webserver started at http://127.24.186.254:33637/ using document root <none> and password file <none>
I20260928 20:29:03.160419 15470 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:03.160467 15470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:03.160600 15470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:03.161715 15470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data/instance:
uuid: "9a821c69448a4808a3fd71c752ca43d2"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "4ce1075a8a283f3ff649fa5fb28f30ae"
server_key_iv: "6521f7c86a68ba4a45fdf039e1813b24"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.162077 15470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance:
uuid: "9a821c69448a4808a3fd71c752ca43d2"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "4ce1075a8a283f3ff649fa5fb28f30ae"
server_key_iv: "6521f7c86a68ba4a45fdf039e1813b24"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.163511 15470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:29:03.164495 15485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:03.164712 15470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:29:03.164825 15470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal
uuid: "9a821c69448a4808a3fd71c752ca43d2"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "4ce1075a8a283f3ff649fa5fb28f30ae"
server_key_iv: "6521f7c86a68ba4a45fdf039e1813b24"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.164913 15470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:03.182566 15470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:03.185760 15470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:03.185930 15470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:03.191052 15470 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.254:35649
I20260928 20:29:03.191087 15537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.254:35649 every 8 connection(s)
I20260928 20:29:03.191609 15470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/data/info.pb
I20260928 20:29:03.192572 15538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:03.195500 15538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.200735 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15470
I20260928 20:29:03.201023 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-0/wal/instance
I20260928 20:29:03.201615 25323 external_mini_cluster.cc:1428] Setting key 66cb2d70a0021515dc63d07598a51a84
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:03.207065 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.196543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35649 (local address 127.0.0.1:57760)
0928 20:29:03.196764 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.196815 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.197394 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.197703 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.197705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.197981 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.198147 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.198150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.199171 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.199173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.199956 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.199958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.200583 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.200586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.200736 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.201538 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.201549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.203414 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.205481 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.205486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.205488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.205694 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.205958 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.205958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.205959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.206057 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.206450 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.206454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.206520 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.206631 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.206872 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":88,"thread_start_us":101,"threads_started":1}
I20260928 20:29:03.207075 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.196582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.24.186.254:35649)
0928 20:29:03.196795 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.196803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.196884 (+ 81us) server_negotiation.cc:407] Connection header received
0928 20:29:03.197563 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.197565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.197583 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.197675 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.198250 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.199069 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.200042 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.200373 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.200469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.203525 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.203532 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.203533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.203539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.205344 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.205796 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.205799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.205800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.205885 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.206144 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.206145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.206145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.206375 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.206473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.206665 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.206807 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":119,"threads_started":1}
I20260928 20:29:03.208215 15538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.208588 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.208463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35114)
0928 20:29:03.208514 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.208562 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:29:03.208745 15486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
I20260928 20:29:03.209187 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.209089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35120)
0928 20:29:03.209141 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.209166 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 20:29:03.209299 15538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:42171: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.24.186.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
Entry for principal kudu/127.24.186.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab.
I20260928 20:29:03.229161 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:42171
--webserver_interface=127.24.186.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:29:03.258561 15538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } attempt: 1
I20260928 20:29:03.259086 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.258946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35122)
0928 20:29:03.259019 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.259059 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 20:29:03.259413 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.259322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35138)
0928 20:29:03.259367 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.259393 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:29:03.259503 15538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:42171: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
W20260928 20:29:03.317466 15545 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:03.317736 15545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:03.317767 15545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:03.317813 15545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:03.319012 15545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:03.319094 15545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:03.319118 15545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:03.319147 15545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:03.319180 15545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:03.320753 15545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.253.keytab
--principal=kudu/127.24.186.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.253:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.24.186.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:03.321049 15545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:03.321281 15545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:29:03.326431 15553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:03.326390 15551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:03.326608 15545 server_base.cc:1034] running on GCE node
W20260928 20:29:03.326390 15550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:03.326917 15545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:03.327275 15545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:03.327641 15538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } attempt: 2
I20260928 20:29:03.328090 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.327912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35148)
0928 20:29:03.328024 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.328060 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 20:29:03.328462 15545 hybrid_clock.cc:648] HybridClock initialized: now 1790627343328431 us; error 45 us; skew 500 ppm
I20260928 20:29:03.328557 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.328437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35164)
0928 20:29:03.328483 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.328509 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 20:29:03.328679 15538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.253:42171: Network error: Client connection negotiation failed: client connection to 127.24.186.253:42171: connect: Connection refused (error 111)
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:03.331046 15545 init.cc:377] Logged in from keytab as kudu/127.24.186.253@KRBTEST.COM (short username kudu)
I20260928 20:29:03.331513 15545 webserver.cc:460] Webserver started at http://127.24.186.253:38761/ using document root <none> and password file <none>
I20260928 20:29:03.331768 15545 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:03.331863 15545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:03.332019 15545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:03.333392 15545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data/instance:
uuid: "50e13da51cfd4001ad0b6d27819b72bd"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "aaeda9e860c76af18f8f4da4436dc769"
server_key_iv: "c6d35ca99eff6687b4337aa6e7cf73ee"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.333824 15545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal/instance:
uuid: "50e13da51cfd4001ad0b6d27819b72bd"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "aaeda9e860c76af18f8f4da4436dc769"
server_key_iv: "c6d35ca99eff6687b4337aa6e7cf73ee"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.335371 15545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 20:29:03.336355 15560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:03.336741 15545 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 20:29:03.336879 15545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal
uuid: "50e13da51cfd4001ad0b6d27819b72bd"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "aaeda9e860c76af18f8f4da4436dc769"
server_key_iv: "c6d35ca99eff6687b4337aa6e7cf73ee"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.336977 15545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:03.351224 15545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:03.353925 15545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:03.354125 15545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:03.358783 15545 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.253:42171
I20260928 20:29:03.358834 15612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.253:42171 every 8 connection(s)
I20260928 20:29:03.359241 15545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/data/info.pb
I20260928 20:29:03.360234 15613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:03.361862 15613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.364723 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15545
I20260928 20:29:03.364877 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-1/wal/instance
I20260928 20:29:03.365077 25323 external_mini_cluster.cc:1428] Setting key 80c783c24aed40dba5a5678e6947ed43
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:03.372893 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.362820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.24.186.254:35649)
0928 20:29:03.362895 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.362897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.363155 (+ 258us) server_negotiation.cc:407] Connection header received
0928 20:29:03.363686 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.363688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.363702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.363748 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.364718 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.365530 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.366567 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.366782 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.366872 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.369431 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.369432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.369433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.369441 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.371224 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.371667 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.371669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.371669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.371735 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.372079 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.372080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.372080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.372241 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.372320 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.372675 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.372764 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 20:29:03.372916 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.362708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35649 (local address 127.0.0.1:57774)
0928 20:29:03.363013 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.363051 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.363578 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.363875 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.363879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.364144 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.364546 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.364557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.365666 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.365667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.366456 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.366458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.366882 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.366883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.367017 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.367612 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.367623 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.369331 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.371323 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.371329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.371331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.371586 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.371881 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.371882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.371884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.372000 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.372337 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.372354 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.372485 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.372601 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.372742 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.24.186.252@KRBTEST.COM; defaulting to no policy
I20260928 20:29:03.373509 15613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.24.186.252@KRBTEST.COM" created.
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:03.381110 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.373770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35180)
0928 20:29:03.373820 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.373831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.373868 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.374146 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.374147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.374295 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.374341 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.374343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.375216 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.375217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.375732 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.375733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.376268 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.376269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.376332 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.376803 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.376809 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.378321 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.380070 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.380071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.380072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.380230 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.380443 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.380443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.380444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.380504 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.380793 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.380794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.380832 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.380892 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.380994 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:29:03.381176 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.373865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.24.186.253:42171)
0928 20:29:03.374015 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.374020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.374036 (+ 16us) server_negotiation.cc:407] Connection header received
0928 20:29:03.374053 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.374054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.374065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.374146 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.374422 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.375128 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.375849 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.376176 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.376270 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.378414 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.378423 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.378424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.378430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.379983 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.380319 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.380323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.380323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.380379 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.380575 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.380576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.380576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.380723 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.380817 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.380930 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.381047 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 20:29:03.381821 15613 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.382170 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.382054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59820)
0928 20:29:03.382099 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.382142 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 20:29:03.382289 15561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
I20260928 20:29:03.382712 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.382620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59836)
0928 20:29:03.382667 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.382690 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 20:29:03.382817 15613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:42517: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
Entry for principal kudu/127.24.186.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab.
I20260928 20:29:03.394210 25323 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9jCLVo/build/release/bin/kudu
/tmp/dist-test-task9jCLVo/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:42517
--webserver_interface=127.24.186.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.24.186.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 20:29:03.419874 15538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } attempt: 3
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:03.427699 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.420368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.24.186.253:42171)
0928 20:29:03.420440 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.420442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.420478 (+ 36us) server_negotiation.cc:407] Connection header received
0928 20:29:03.420502 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.420503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.420512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.420554 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.420912 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.421408 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.422404 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.422620 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.422674 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.424749 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.424750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.424751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.424758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.426161 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.426665 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.426666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.426666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.426725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.426978 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.426979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.426980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.427108 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.427169 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.427555 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.427633 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:29:03.427696 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.420298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35194)
0928 20:29:03.420359 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.420373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.420436 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.420578 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.420578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.420763 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.420837 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.420839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.421486 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.421487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.422313 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.422316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.422695 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.422696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.422768 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.423230 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.423236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.424660 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.426246 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.426247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.426247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.426463 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.426805 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.426806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.426807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.426885 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.427324 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.427325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.427381 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.427477 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.427590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:29:03.428310 15538 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.428675 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.428561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59844)
0928 20:29:03.428614 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.428649 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 20:29:03.428987 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.428903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59858)
0928 20:29:03.428947 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.428971 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:29:03.429096 15538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:42517: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
I20260928 20:29:03.432015 15613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } attempt: 1
I20260928 20:29:03.432391 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.432268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59870)
0928 20:29:03.432332 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.432367 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 20:29:03.432719 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.432640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59884)
0928 20:29:03.432681 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.432705 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 20:29:03.432837 15613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:42517: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
I20260928 20:29:03.460294 15538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } attempt: 1
I20260928 20:29:03.460947 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.460803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59898)
0928 20:29:03.460875 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.460916 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 20:29:03.461380 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.461252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59906)
0928 20:29:03.461326 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.461357 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 20:29:03.461535 15538 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:42517: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
W20260928 20:29:03.481735 15620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 20:29:03.481953 15620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 20:29:03.481972 15620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 20:29:03.481993 15620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 20:29:03.483106 15620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 20:29:03.483146 15620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 20:29:03.483161 15620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 20:29:03.483174 15620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 20:29:03.483186 15620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 20:29:03.484725 15620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.24.186.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.24.186.254:35649,127.24.186.253:42171,127.24.186.252:42517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/krb5kdc/kudu_127.24.186.252.keytab
--principal=kudu/127.24.186.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.24.186.252:42517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.24.186.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 19:10:17 UTC on ccycloud.quasar-awmulw.root.comops.site
build id 4677981
I20260928 20:29:03.484930 15620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 20:29:03.485106 15620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 20:29:03.489926 15628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:03.489933 15626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 20:29:03.489925 15625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 20:29:03.490041 15620 server_base.cc:1034] running on GCE node
I20260928 20:29:03.490424 15620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 20:29:03.490727 15620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 20:29:03.491868 15620 hybrid_clock.cc:648] HybridClock initialized: now 1790627343491860 us; error 35 us; skew 500 ppm
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 20:29:03.494232 15620 init.cc:377] Logged in from keytab as kudu/127.24.186.252@KRBTEST.COM (short username kudu)
I20260928 20:29:03.494665 15620 webserver.cc:460] Webserver started at http://127.24.186.252:33601/ using document root <none> and password file <none>
I20260928 20:29:03.494917 15620 fs_manager.cc:359] Metadata directory not provided
I20260928 20:29:03.494988 15620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 20:29:03.495141 15620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 20:29:03.496322 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data/instance:
uuid: "aaad397df26e49a2ac29d5a96aaf9ee4"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "0e0e9a8eca21449952d2a070d962187f"
server_key_iv: "60d467c83d1b67c58c959ed050e55f4e"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.496701 15620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal/instance:
uuid: "aaad397df26e49a2ac29d5a96aaf9ee4"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "0e0e9a8eca21449952d2a070d962187f"
server_key_iv: "60d467c83d1b67c58c959ed050e55f4e"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.498100 15620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 20:29:03.498896 15635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 20:29:03.499145 15620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 20:29:03.499269 15620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data,/tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal
uuid: "aaad397df26e49a2ac29d5a96aaf9ee4"
format_stamp: "Formatted at 2026-09-28 20:29:03 on dist-test-slave-k477"
server_key: "0e0e9a8eca21449952d2a070d962187f"
server_key_iv: "60d467c83d1b67c58c959ed050e55f4e"
server_key_version: "encryptionkey@0"
I20260928 20:29:03.499349 15620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 20:29:03.500984 15613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } attempt: 2
I20260928 20:29:03.501462 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.501302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59918)
0928 20:29:03.501370 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.501412 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 20:29:03.501863 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.501756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59928)
0928 20:29:03.501802 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.501833 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 20:29:03.501967 15613 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.24.186.252:42517: Network error: Client connection negotiation failed: client connection to 127.24.186.252:42517: connect: Connection refused (error 111)
I20260928 20:29:03.516595 15620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 20:29:03.518438 15620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 20:29:03.518589 15620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 20:29:03.522429 15620 rpc_server.cc:307] RPC server started. Bound to: 127.24.186.252:42517
I20260928 20:29:03.522470 15687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.24.186.252:42517 every 8 connection(s)
I20260928 20:29:03.522851 15620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/data/info.pb
I20260928 20:29:03.523756 15688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 20:29:03.525291 15688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.529700 25323 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9jCLVo/build/release/bin/kudu as pid 15620
I20260928 20:29:03.529831 25323 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9jCLVo/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790627060930553-25323-0/minicluster-data/master-2/wal/instance
I20260928 20:29:03.530035 25323 external_mini_cluster.cc:1428] Setting key 2424b0a4e00b6eb378f88a5af3483255
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:03.534616 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.526027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.24.186.254:35649)
0928 20:29:03.526097 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.526099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.526316 (+ 217us) server_negotiation.cc:407] Connection header received
0928 20:29:03.526787 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.526788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.526798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.526859 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.527457 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.527995 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.528896 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.529079 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.529134 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.531526 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.531528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.531528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.531534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.533210 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.533630 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.533632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.533632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.533697 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.533949 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.533950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.533951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.534074 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.534150 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.534429 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.534512 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 20:29:03.534816 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.525981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.254:35649 (local address 127.0.0.1:57790)
0928 20:29:03.526202 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.526237 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.526680 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.526931 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.526934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.527179 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.527369 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.527372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.528067 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.528068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.528818 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.528821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.529153 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.529153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.529252 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.529750 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.529760 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.531435 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.533304 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.533310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.533312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.533537 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.533762 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.533763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.533764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.533870 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.534142 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.534149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.534290 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.534407 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.534566 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":123,"threads_started":1}
I20260928 20:29:03.535712 15688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } has no permanent_uuid. Determining permanent_uuid...
I20260928 20:29:03.536685 15538 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } attempt: 2
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.254@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:29:03.541213 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.530438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.24.186.252:42517)
0928 20:29:03.530603 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.530609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.531253 (+ 644us) server_negotiation.cc:407] Connection header received
0928 20:29:03.531277 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.531278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.531300 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.531363 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.532153 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.533092 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.533837 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.534417 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.534619 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.537911 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.537921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.537921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.537928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.539904 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.540327 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.540331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.540331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.540395 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.540643 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.540643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.540644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.540772 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.540965 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.541009 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.541091 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:03.545698 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.536012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.253:42171 (local address 127.0.0.1:35200)
0928 20:29:03.536063 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.536075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.536114 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.536425 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.536427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.536574 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.536632 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.536633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.537688 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.537689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.538269 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.538271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.539431 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.539433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.539513 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.539981 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.539986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.542348 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.544638 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.544639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.544640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.544847 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.545061 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.545062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.545062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.545130 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.545399 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.545400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.545440 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.545502 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.545598 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 20:29:03.545784 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.536937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59940)
0928 20:29:03.536981 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.536990 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.537024 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.537293 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.537294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.537453 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.537496 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.537498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.538350 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.538351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.538877 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.538879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.539319 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.539320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.539379 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.539775 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.539780 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.542041 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.544389 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.544390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.544390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.544609 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.545054 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.545055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.545055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.545150 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.545477 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.545478 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.545522 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.545591 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.545675 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 20:29:03.545965 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.536986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.24.186.252:42517)
0928 20:29:03.537163 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.537165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.537172 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:29:03.537190 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.537191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.537200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.537264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.537571 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.538260 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.538934 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.539239 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.539322 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.542129 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.542131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.542131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.542137 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.544282 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.544672 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.544673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.544673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.544724 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.545233 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.545234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.545235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.545378 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.545489 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.545629 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.545720 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260928 20:29:03.546115 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.536199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.24.186.253:42171)
0928 20:29:03.536269 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.536271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.536281 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:29:03.536310 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.536312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.536325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.536401 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.536852 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.537587 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.538978 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.539278 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.539366 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.542451 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.542453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.542454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.542462 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.544558 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.544931 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.544932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.544932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.544986 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.545200 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.545201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.545201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.545320 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.545394 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.545884 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.545991 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 20:29:03.546711 15688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.254@KRBTEST.COM
I20260928 20:29:03.548161 15538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: Bootstrap starting.
I20260928 20:29:03.548947 15538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:03.549304 15538 log.cc:824] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:03.550073 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.542024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.24.186.254:35649)
0928 20:29:03.542065 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.542067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.542075 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:29:03.542095 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.542096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.542105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.542168 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.542741 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.543196 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.544163 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.544481 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.544554 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.547357 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.547359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.547359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.547366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.548785 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.549178 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.549179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.549179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.549238 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.549468 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.549469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.549470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.549609 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.549732 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.549930 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.549995 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:29:03.550184 15538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: No bootstrap required, opened a new log
I20260928 20:29:03.551858 15538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.552007 15538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a821c69448a4808a3fd71c752ca43d2, State: Initialized, Role: FOLLOWER
I20260928 20:29:03.552096 15538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.552397 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.552451 15538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:03.552503 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.252@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
W20260928 20:29:03.554970 15706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:03.555373 15706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.24.186.253@KRBTEST.COM
I20260928 20:29:03.557413 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.546988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59950)
0928 20:29:03.547030 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.547039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.547072 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.547198 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.547199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.547353 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.547399 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.547401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.548790 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.548791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.549545 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.549548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.552340 (+ 2792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.552341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.552398 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.553013 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.553018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.554445 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.556316 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.556317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.556318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.556513 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.556683 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.556684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.556684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.556732 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.557088 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.557089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.557144 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.557214 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.557312 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 20:29:03.557653 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.547024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.24.186.252:42517)
0928 20:29:03.547060 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.547062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.547072 (+ 10us) server_negotiation.cc:407] Connection header received
0928 20:29:03.547133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.547133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.547144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.547196 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.547498 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.548697 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.549633 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.552145 (+ 2512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.552836 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.554525 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.554526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.554526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.554532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.556228 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.556580 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.556581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.556581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.556626 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.556786 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.556787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.556788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.556993 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.557058 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.557519 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.557585 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 20:29:03.555282 15470 master_runner.cc:186] Error getting master registration for 127.24.186.253:42171: OK, instance_id { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" instance_seqno: 1790627343357189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:29:03.557897 15470 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 20:29:03.559026 15688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: Bootstrap starting.
I20260928 20:29:03.559768 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.550803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.24.186.253:42171)
0928 20:29:03.550849 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.550851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.550860 (+ 9us) server_negotiation.cc:407] Connection header received
0928 20:29:03.550880 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.550881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.550890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.550974 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.551425 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.551937 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.552753 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.553082 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.553139 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.556493 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.556495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.556495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.556504 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.558283 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.558699 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.558701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.558701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.558764 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.559001 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.559002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.559002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.559140 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.559295 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.559593 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.559672 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 20:29:03.559841 15688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:03.560155 15688 log.cc:824] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:03.561017 15688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: No bootstrap required, opened a new log
I20260928 20:29:03.562040 25323 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 20:29:03.562831 15512 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a821c69448a4808a3fd71c752ca43d2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57796
I20260928 20:29:03.563002 15512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 20:29:03.563056 15512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:03.563095 15688 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.563259 15688 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaad397df26e49a2ac29d5a96aaf9ee4, State: Initialized, Role: FOLLOWER
I20260928 20:29:03.563352 15688 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.563710 15688 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:03.563663 15709 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.563910 15709 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:03.564131 15512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.564394 15512 leader_election.cc:288] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [CANDIDATE]: Term 1 election: Requested vote from peers 50e13da51cfd4001ad0b6d27819b72bd (127.24.186.253:42171), aaad397df26e49a2ac29d5a96aaf9ee4 (127.24.186.252:42517)
I20260928 20:29:03.565888 15662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a821c69448a4808a3fd71c752ca43d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4"
I20260928 20:29:03.566007 15662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:03.567665 15662 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c69448a4808a3fd71c752ca43d2 in term 1.
I20260928 20:29:03.567903 15486 leader_election.cc:302] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a821c69448a4808a3fd71c752ca43d2, aaad397df26e49a2ac29d5a96aaf9ee4; no voters:
I20260928 20:29:03.568078 15695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 20:29:03.568269 15695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9a821c69448a4808a3fd71c752ca43d2, State: Running, Role: LEADER
I20260928 20:29:03.568372 15695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.567516 15620 master_runner.cc:186] Error getting master registration for 127.24.186.253:42171: OK, instance_id { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" instance_seqno: 1790627343357189 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 20:29:03.570683 15620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 20:29:03.572055 15720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:03.572129 15720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:29:03.573375 15726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a821c69448a4808a3fd71c752ca43d2. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.573534 15726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: This master's current role is: LEADER
I20260928 20:29:03.577479 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.565966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.24.186.254:35649)
0928 20:29:03.566055 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.566057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.568911 (+ 2854us) server_negotiation.cc:407] Connection header received
0928 20:29:03.569012 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.569013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.569023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.569068 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.569966 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.570748 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.571950 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.572253 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.572293 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.574333 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.574335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.574335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.574344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.576088 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.576507 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.576508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.576508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.576576 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.576800 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.576801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.576801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.576978 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.577065 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.577297 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.577380 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:29:03.579497 15731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 20:29:03.580174 15731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 20:29:03.582219 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.568916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.24.186.252:42517)
0928 20:29:03.568951 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.568953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.569907 (+ 954us) server_negotiation.cc:407] Connection header received
0928 20:29:03.570785 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.570786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.570796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.570826 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.573668 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.574582 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.575617 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.575939 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.575986 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.578952 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.578953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.578954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.578962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.580846 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.581336 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.581337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.581337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.581393 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.581605 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.581606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.581606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.581747 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.581830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.582036 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.582119 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 20:29:03.582582 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.568572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.24.186.253:42171)
0928 20:29:03.568619 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.568622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.569707 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 20:29:03.569954 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.569955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.569968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.569998 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.571607 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.572350 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.576349 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.576644 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.576692 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.578924 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.578925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.578926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.578933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.580622 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.581455 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.581456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.581456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.581519 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.581731 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.581733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.581733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.581873 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.581962 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.582398 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.582486 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 20:29:03.584195 15662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [term 1 FOLLOWER]: Refusing update from remote peer 9a821c69448a4808a3fd71c752ca43d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:29:03.584563 15726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:29:03.586417 15709 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a821c69448a4808a3fd71c752ca43d2. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.586534 15709 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:03.588759 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.588864 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: This master's current role is: LEADER
I20260928 20:29:03.588968 15709 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.589057 15709 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:03.589267 15733 mvcc.cc:205] Tried to move back new op lower bound from 7334409599317856256 to 7334409599256846336. Current Snapshot: MvccSnapshot[applied={T|T < 7334409599317856256}]
I20260928 20:29:03.592583 15731 catalog_manager.cc:1347] Generated new cluster ID: df3b8125c43f47b99515f8daf369d902
I20260928 20:29:03.592641 15731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 20:29:03.593110 15613 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } attempt: 3
Sep 28 20:29:03 dist-test-slave-k477 krb5kdc[15457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790627343, etypes {rep=17 tkt=17 ses=17}, kudu/127.24.186.253@KRBTEST.COM for kudu/127.24.186.252@KRBTEST.COM
I20260928 20:29:03.604257 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.593384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.24.186.252:42517 (local address 127.0.0.1:59964)
0928 20:29:03.593475 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 20:29:03.593502 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 20:29:03.593544 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 20:29:03.594681 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 20:29:03.594683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 20:29:03.594883 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 20:29:03.594933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.594935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.595932 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.595934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.596772 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 20:29:03.596775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.597309 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.597310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 20:29:03.597367 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.597885 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 20:29:03.597890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 20:29:03.600363 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 20:29:03.602897 (+ 2534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.602898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.602899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.603144 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.603463 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 20:29:03.603464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 20:29:03.603464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 20:29:03.603527 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 20:29:03.603909 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 20:29:03.603910 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 20:29:03.603959 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 20:29:03.604022 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 20:29:03.604149 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 20:29:03.604643 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:03.593618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.24.186.252:42517)
0928 20:29:03.594539 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.594541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.594548 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:29:03.594569 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.594570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.594581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.594630 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.595012 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.595767 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.596887 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:03.597196 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:03.597252 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:03.600455 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 20:29:03.600456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 20:29:03.600457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 20:29:03.600464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 20:29:03.602796 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.603226 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.603227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.603228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.603306 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 20:29:03.603590 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 20:29:03.603591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 20:29:03.603591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 20:29:03.603821 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 20:29:03.603883 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:03.604475 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 20:29:03.604555 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888}
I20260928 20:29:03.606202 15613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: Bootstrap starting.
I20260928 20:29:03.606916 15613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: Neither blocks nor log segments found. Creating new log.
I20260928 20:29:03.607241 15613 log.cc:824] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: Log is configured to *not* fsync() on all Append() calls
I20260928 20:29:03.608618 15731 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 20:29:03.609944 15731 catalog_manager.cc:1504] Loading token signing keys...
I20260928 20:29:03.612627 15613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: No bootstrap required, opened a new log
I20260928 20:29:03.614740 15613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.614907 15613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e13da51cfd4001ad0b6d27819b72bd, State: Initialized, Role: FOLLOWER
I20260928 20:29:03.614989 15613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } }
I20260928 20:29:03.615346 15613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 20:29:03.615437 15586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 20:29:03.616132 15587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a821c69448a4808a3fd71c752ca43d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50e13da51cfd4001ad0b6d27819b72bd"
I20260928 20:29:03.616834 15586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [term 1 FOLLOWER]: Refusing update from remote peer 9a821c69448a4808a3fd71c752ca43d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 20:29:03.617214 15726 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 20:29:03.618072 15587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a821c69448a4808a3fd71c752ca43d2 in term 1.
I20260928 20:29:03.619832 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.619920 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:03.619969 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a821c69448a4808a3fd71c752ca43d2. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.620046 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 20:29:03.620085 15738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.620148 15738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd [sys.catalog]: This master's current role is: FOLLOWER
W20260928 20:29:03.620723 15749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 20:29:03.620782 15749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 20:29:03.625801 15731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2: Generated new TSK 0
I20260928 20:29:03.625988 15731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 20:29:03.628470 15751 mvcc.cc:205] Tried to move back new op lower bound from 7334409599410544640 to 7334409599256846336. Current Snapshot: MvccSnapshot[applied={T|T < 7334409599317856256}]
I20260928 20:29:03.632761 15695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a821c69448a4808a3fd71c752ca43d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a821c69448a4808a3fd71c752ca43d2" member_type: VOTER last_known_addr { host: "127.24.186.254" port: 35649 } } peers { permanent_uuid: "50e13da51cfd4001ad0b6d27819b72bd" member_type: VOTER last_known_addr { host: "127.24.186.253" port: 42171 } } peers { permanent_uuid: "aaad397df26e49a2ac29d5a96aaf9ee4" member_type: VOTER last_known_addr { host: "127.24.186.252" port: 42517 } } }
I20260928 20:29:03.632879 15695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a821c69448a4808a3fd71c752ca43d2 [sys.catalog]: This master's current role is: LEADER
W20260928 20:29:03.647040 15541 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.645409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.24.186.254:35649)
0928 20:29:03.645451 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.645453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.646359 (+ 906us) server_negotiation.cc:407] Connection header received
0928 20:29:03.646466 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.646467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.646479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.646511 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.646965 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57812: BlockingRecv error: recv got EOF from 127.0.0.1:57812 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:29:03.647773 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35224 (local address 127.24.186.253:42171)
0928 20:29:03.645856 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.645858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.647289 (+ 1431us) server_negotiation.cc:407] Connection header received
0928 20:29:03.647384 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.647385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.647395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.647425 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.647716 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35224: BlockingRecv error: recv got EOF from 127.0.0.1:35224 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:29:03.648528 15693 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.646099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.24.186.252:42517)
0928 20:29:03.646151 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.646153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.647997 (+ 1844us) server_negotiation.cc:407] Connection header received
0928 20:29:03.648132 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.648133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.648143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.648174 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.648465 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59974: BlockingRecv error: recv got EOF from 127.0.0.1:59974 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 20:29:03.650635 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.649870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.24.186.253:42171)
0928 20:29:03.649905 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.649906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.650116 (+ 210us) server_negotiation.cc:407] Connection header received
0928 20:29:03.650234 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.650236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.650255 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.650285 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.650588 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35234: BlockingRecv error: recv got EOF from 127.0.0.1:35234 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 20:29:03.661115 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.654239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.24.186.253:42171)
0928 20:29:03.654272 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.654274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.660474 (+ 6200us) server_negotiation.cc:407] Connection header received
0928 20:29:03.660615 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.660616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.660628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.660659 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.661056 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35244: BlockingRecv error: recv got EOF from 127.0.0.1:35244 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 20:29:03.665248 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.664370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.24.186.253:42171)
0928 20:29:03.664408 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.664410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.664683 (+ 273us) server_negotiation.cc:407] Connection header received
0928 20:29:03.664789 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.664790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.664801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.664836 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.665194 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35260: BlockingRecv error: recv got EOF from 127.0.0.1:35260 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 20:29:03.671492 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.670634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.24.186.253:42171)
0928 20:29:03.670676 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.670678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.670909 (+ 231us) server_negotiation.cc:407] Connection header received
0928 20:29:03.671016 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.671017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.671029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.671059 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.671418 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35270: BlockingRecv error: recv got EOF from 127.0.0.1:35270 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 20:29:03.681816 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.680877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.24.186.253:42171)
0928 20:29:03.680921 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.680923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.681177 (+ 254us) server_negotiation.cc:407] Connection header received
0928 20:29:03.681286 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.681287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.681300 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.681332 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.681751 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35284: BlockingRecv error: recv got EOF from 127.0.0.1:35284 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:03.700239 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.699262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.24.186.253:42171)
0928 20:29:03.699310 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.699313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.699571 (+ 258us) server_negotiation.cc:407] Connection header received
0928 20:29:03.699677 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.699678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.699692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.699723 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.700171 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35286: BlockingRecv error: recv got EOF from 127.0.0.1:35286 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 20:29:03.734544 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.733680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.24.186.253:42171)
0928 20:29:03.733724 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.733726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.733948 (+ 222us) server_negotiation.cc:407] Connection header received
0928 20:29:03.734047 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.734048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.734059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.734089 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.734465 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 20:29:03.800930 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.800079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.24.186.253:42171)
0928 20:29:03.800138 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.800141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.800271 (+ 130us) server_negotiation.cc:407] Connection header received
0928 20:29:03.800347 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.800348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.800359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.800410 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.800839 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35292: BlockingRecv error: recv got EOF from 127.0.0.1:35292 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 20:29:03.930987 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:03.930204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.24.186.253:42171)
0928 20:29:03.930297 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:03.930300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:03.930401 (+ 101us) server_negotiation.cc:407] Connection header received
0928 20:29:03.930453 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:03.930454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:03.930465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:03.930512 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:03.930899 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35298: BlockingRecv error: recv got EOF from 127.0.0.1:35298 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 20:29:04.189498 15617 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 20:29:04.188784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.24.186.253:42171)
0928 20:29:04.188857 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.188859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.188890 (+ 31us) server_negotiation.cc:407] Connection header received
0928 20:29:04.189006 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.189009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.189025 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.189096 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 20:29:04.189449 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35308: BlockingRecv error: recv got EOF from 127.0.0.1:35308 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 20:29:04.572652 15720 catalog_manager.cc:1259] Loaded cluster ID: df3b8125c43f47b99515f8daf369d902
I20260928 20:29:04.572736 15720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: loading cluster ID for follower catalog manager: success
I20260928 20:29:04.573518 15720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: acquiring CA information for follower catalog manager: success
I20260928 20:29:04.573788 15720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aaad397df26e49a2ac29d5a96aaf9ee4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:29:04.621264 15749 catalog_manager.cc:1259] Loaded cluster ID: df3b8125c43f47b99515f8daf369d902
I20260928 20:29:04.621351 15749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: loading cluster ID for follower catalog manager: success
I20260928 20:29:04.622161 15749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: acquiring CA information for follower catalog manager: success
I20260928 20:29:04.622442 15749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50e13da51cfd4001ad0b6d27819b72bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 20:29:04.706435 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:04.703045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.24.186.253:42171)
0928 20:29:04.703249 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.703252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.703418 (+ 166us) server_negotiation.cc:407] Connection header received
0928 20:29:04.703546 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.703547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.703634 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.703894 (+ 260us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:04.704114 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.704734 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.705589 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.705799 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.705877 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:04.706061 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:04.706181 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:04.706276 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:04.706299 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 20:29:04.706326 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260928 20:29:04.710453 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:04.708203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.24.186.252:42517)
0928 20:29:04.708295 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.708297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.708305 (+ 8us) server_negotiation.cc:407] Connection header received
0928 20:29:04.708367 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.708368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.708403 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.708496 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:04.708572 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.709029 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.709757 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.709952 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.710015 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:04.710116 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:04.710214 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:04.710266 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:04.710299 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 20:29:04.710316 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":50,"threads_started":1}
I20260928 20:29:04.715219 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:04.712286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.24.186.253:42171)
0928 20:29:04.712327 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.712329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.712455 (+ 126us) server_negotiation.cc:407] Connection header received
0928 20:29:04.712554 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.712555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.712569 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.712630 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:04.712796 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.713552 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.714377 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.714641 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.714718 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:04.714881 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:04.714915 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:04.714969 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:04.715100 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 20:29:04.715127 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 20:29:04.715516 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:04.712533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60000 (local address 127.24.186.252:42517)
0928 20:29:04.712568 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.712570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.712684 (+ 114us) server_negotiation.cc:407] Connection header received
0928 20:29:04.712752 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.712754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.712764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.712808 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:04.712969 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.713669 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.714679 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.715004 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.715068 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:04.715243 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:04.715281 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:04.715311 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:04.715423 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 20:29:04.715446 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 20:29:04.715725 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 20:29:04.712256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.24.186.254:35649)
0928 20:29:04.712576 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0928 20:29:04.712578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 20:29:04.712585 (+ 7us) server_negotiation.cc:407] Connection header received
0928 20:29:04.712631 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 20:29:04.712632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 20:29:04.712685 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 20:29:04.712935 (+ 250us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 20:29:04.712974 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.713462 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.714631 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 20:29:04.714954 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 20:29:04.715046 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 20:29:04.715239 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 20:29:04.715359 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 20:29:04.715594 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 20:29:04.715614 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 20:29:04.715644 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":49,"threads_started":1}
I20260928 20:29:04.717567 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15470
I20260928 20:29:04.723292 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15545
I20260928 20:29:04.729725 25323 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9jCLVo/build/release/bin/kudu with pid 15620
2026-09-28T20:29:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2881 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5860 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283806 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST